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ABSTRACT 

The goal of this study is to develop a set of empirically and theoretically sound 
citation-based bibliometric indicators of scientific research performance and apply them 
in an exploratory comparative study of the University of Tennessee-Knoxville’s Nutrition 
Department with three of its peer programs at the University of Florida, the University of 
Georgia, and Virginia Polytechnic Institute and State University over a five-year period 
(1992-1996). A search of the bibliometric literature revealed much criticism of the use of 
citation analysis, little consensus about the solutions, and even less empirical data 
applicable to departmental level studies. Therefore for the purpose of this study, self- 
citations were not excluded from the data, equal publication credit was given for multi- 
authored papers, and the two-year citation window was used in the calculation of 
bibliometric indicators and impact measures. The overall conceptual approach used was 
a limited version of Martin and Irvine’s (1981, 1983) methodology of multiple 
converging indicators of scientific performance, using only bibliometric indicators, 
drawing on the citation data from the Institute for Scientific Information’s Science 
Citation Index- Expanded, and the Journal Citation Reports. A set of eight empirically 
sound and theoretically justifiable indicators was developed and applied in this study. Of 
the four peer Nutrition programs evaluated, the UTK Nutrition Department ranked 
second in both the overall rankings and in the Actual Impact Group of indicators, and 
ranked third in the Publication Output and Benchmark Groups of indicators. 
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CHAPTER 1 



INTRODUCTION 

Background 

The goal of the College of Human Ecology at the University of Tennessee- 
Knoxville (UTK) is to become one of the top twenty-five of its kind in the country. To 
this end, Dr. Jim Moran, Dean of the College of Human Ecology, asked Dr. Elizabeth 
Aversa, the Director of the School of Information Sciences, to help him devise 
bibliometric measures for evaluating his component departments and comparing them to 
their peers nationwide (J. Moran, personal communication, June 23, 2000). A preliminary 
bibliometric study of the Nutrition Department, one of the units in UTK’s College of 
Human Ecology, was conducted during the Summer Term of 2000, and forms the basis of 
this study (Ackermann 2000). Nutrition is a science-based discipline focused on 
understanding the relationships among nutrients, between diet and certain diseases such as 
cancer and obesity, and how this understanding can be used to promote healthy living 
(Moran 2000). As a science-based discipline, the primary research publication product for 
the Nutrition field is the scholarly scientific journal article (Van Raan and van Leeuwen 
1998). 
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Research Problem 

“My goal is simple. It is the complete understanding of the universe, 
why it is as it is and why it exists at all.” - Stephen Hawking 
(Institute for Scientific Information 2000; KJhalid, 2000) 

The goal of this study is to develop empirically supported and theoretically 
justifiable publication- and citation-based bibliometric measures for the comparative 
evaluation of scientific research performance of the UTK Nutrition Department and three 
of its counterparts from peer, benchmark institutions as well as to the Nutrition field as a 
whole. The emphasis of this study is on interdepartmental productivity comparisons 
between departmental units rather than on intradepartmental comparisons of individual 
performance, which are suitable for internal purposes such as tenure review and 
promotion. Though the data for these measures will be drawn from the print journal 
publication efforts of a program’s research faculty over a five-year period (1992-1996), it 
is the Nutrition Department itself, not the individual faculty researcher, that will be the 
basic unit of comparison (cf Kim and Kim 2000; Cronin and Overfelt 1994). Once 
completed, this study will provide some of the baseline bibliometric data for the ongoing 
comparison and ranking of the UTK Nutrition Department’s current and future research 
performance with similar departments at peer institutions as well as with the entire 
Nutrition discipline. 
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Terminology 

For the purposes of this study, bibliometrics or bibliometry refers to "the 
measurement of scientific publications and of their impact on the scientific community, 
assessed by the citations they attract,” providing a “portfolio of indicators that can be 
combined to give a useful picture of recent research activity" (Garfield 1989, 94). In this 
context, a bibliometric indicator is a partial or incomplete measure of a specific aspect of 
scholarly scientific publishing (Martin and Irvine 1983; Martin 1996), and is sometimes 
used interchangeably with bibliometric measure or bibliometric method in the literature 
(e.g., Garfield 1989). A citation (or cite) is when one document (the citing document) 
formally references (or cites) another document (the cited document) as an information 
source, or to support a position or interpretation, etc. (Institute for Scientific Information 
1999; MacRoberts and MacRoberts 1989). A self-citation (or self-cite) is when the citing 
document and the cited document share one or more of the same authors (Phelan 1999; 
Noyons, Moed, and Luwel 1999), and a document’s citedness is the number of times it 
has been cited by other citing documents. Papers or articles are original journal research 
articles, technical notes, and reviews articles, excluding letters, editorials, meeting 
abstracts, and news items (Institute for Scientific Information 1999). Citation lag is the 
time between an article’s publication in a scholarly print journal and when it receives its 
maximum number of citations (the citation peak). Fast moving fields can have a relatively 
short citation lag measured in months, while for slower moving fields the citation peak can 
be at three to five years (or more) after publication (Garfield 1989). 
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Research refers to both basic “curiosity driven” and long-term, specific application 
or goal oriented “strategic” research (Rinia, van Leeuwen, van Vuren, and van Raan 
1998), while science includes natural science, mathematics, as well as engineering (e.g., 
electronics) and multidisciplinary fields such as nutrition and food. Academic department 
or departments include not only those discipline-specific organizational units traditionally 
associated with a particular university, such as the Department of Physics at the University 
of Tennessee, but also research centers, academic institutes, or research groups, often 
multi-disciplinary, affiliated with a specific university or consortium of universities, such as 
the Inter-university Centre for Micro-Electronics in Leuven, Belgium and the Max-Planck- 
Institut fur Radioastronomie in Bonn, Germany. Research performance and research 
productivity both refer to the published output or product of research activity, as distinct 
from measuring “scientific progress” (Moravcsik 1973; cf Martin and Irvine 1983) which 
is beyond the scope of this study. Research program is often used synonymously with 
academic department, research center, and research group. Unless otherwise noted, a 
field or research field synonymous with a Subject Category from the Institute for 
Scientific Information’s (}.SY s) Journal Citation Reports (JCRs) that contains at least one 
journal that published at least one paper authored or co-authored by at least one member 
of one of the four departments under study. The Nutrition field is composed of the 
Nutrition and Dietetic, and the Biochemistry and Molecular Biology subject categories 
(Moran 2000). 
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In reference to the findings, actual results are those achieved by the members of a 
group under study. Expected results are those that the unit under study would achieve if it 
were performing at the contemporary world or world-wide level (Bourke and Butler 
1995). Since the approximately 80% of the source data for the ISI’s Science Citation 
Index is from the results of Western scientific research (Moed, de Bruin, and van Leeuwen 
1995), world or world-wide in reality refers to the research journal publications from 
Western countries such as the United States, Japan, and Australia (Moed, de Bruin, and 
van Leeuwen 1995; van Raan 1996; van Raan and van Leeuwen 1998). Note that in the 
literature, world and world-wide can be used to refer to other European countries as well, 
though it is not always possible to tell from the context which sense is meant. 

Scope and Limits 

Ideally, a study of research productivity and performance at the departmental level 
would include not only bibliometric indicators, but input measures such as the number of 
researchers, amount of time each researcher can spend actually doing research, the amount 
and sources of research funding, and the number of support staff (Martin and Irvine 1983; 
Koenig 1983; Zachos 1991). The assessment would also include other performance 
output indicators such as peer review results, number of ‘discoveries’ or major scientific 
contributions, and formal recognition in the form of prizes, medals, membership in 
prestigious scientific academies, patents, etc. (Myers 1970; Martin and Irvine 1981, 1983; 
Garfield 1989; Clark 1957; Koenig 1983). 
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The measures in this study however will be limited to publication and citation 
based bibliometric indicators, and the methods to basic citation analysis. Publication 
counts are a measure of research productivity (Rinia et al. 1998; van Raan and van 
Leeuwen 1998: Noyons, Moed, and Luwel 1999; Martin & Irvine, 1983). Citations are 
measures of “quality” in terms of impact or influence on subsequent publications (Rinia, et 
al., 1998; van Raan & van Leeuwen, 1998; Martin & Irvine, 1983). From these two 
measures many other bibliometric measures such as Citations per Publication (CPP) (Rinia 
et al. 1998; van Raan and van Leeuwen 1998; Martin and Irvine 1983; Korevaar and 
Moed 1996), Citation Productivity per Researcher (CPR) (Zachos 1991; Nederhof, 

Meijer, Moed, and van Raan 1993) are derived (for more examples see Appendix B). 
Publication counts and citations are also widely used due in a large part to the relative ease 
of accessing the information from the ISI’s citation indexes, especially through the online 
version Web. of Science. 

The data for this study will be limited to any publication and citation data from 
scholarly, referred articles appearing in the print journal literature, and authored by at least 
one researcher from one of the four nutrition departments being examined. These 
departments are the Nutrition Department at the University of Tennessee- Knoxville, the 
Department of Food Science and Human Nutrition at the University of Florida, the 
Department of Foods and Nutrition at the University of Georgia; and the Department of 
Human Nutrition, Foods, and Exercise at the Virginia Polytechnic Institute and State 
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University. The publication and citation data will be from 1992-1996, and the journal 
impact data from 1994-1998. 

For the purposes of this study, citation analysis is a method of using publication 
output (or the number of publications) and the number of citations received by those 
publications in subsequent publications to “assess the research performance of countries 
[e.g. Bourke and Butler 1993; Pestana 1992], universities [e.g., Davis and Royle 1996; 
Nederhof et al. 1993], departments [e.g., Martin and Irvine 1983,Zachos 1991], or 
persons [e.g., Kim and Kim 2000; Cronin and Overfelt 1994; de Arenas, Valles, Arenas 
1999]” (Noyons, Moed, and Luwel 1999, 115). This study will focus on using citation 
analysis to examine the research performance of academic science departments, 
specifically in the Nutrition field. 

It should be noted in passing that there are other bibliometric methods available for 
studying the scientific literature, such as co-citation analysis, co-word analysis, or citation 
(or scientific literature) mapping. Though useful and highly informative (e.g., Noyons, 
Moed, and Luwel 1999), they are very labor and time intensive which unfortunately puts 
them beyond the scope of this study. (For an excellent treatment of these techniques, see 
Small 1999 and White and McCain 1998). 
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CHAPTER 2 

REVIEW OF THE LITERATURE 
Background 

The continuing reduction of funding available for research has led to a growing 
need for more selectivity in the allocation of resources. Measures of past performance, 
though providing no guarantees, are still some of the most reliable means for deciding 
between competing social and scientific interests. Recently, policy makers and research 
managers have shown a growing interest in the use of indicators for assessing scientific 
output (Garfield 1989; Martin and Irvine 1983), and promoting research quality (van 
Raan 1996). Ideally, these indicators can be generated routinely (Garfield 1989; Martin 
1996) and derived from criteria originating within the scientific community (Martin and 
Irvine 1983). This administrative interest is driven by both the need for greater 
accountability and the increasing criticism of the peer review system (Garfield 1989; 
Martin 1996). 



Need for Funding Selectivity 

Before the 1960s, funding was generally unrestricted with allocation primarily 
based on criteria established by the scientific community and evaluated by the peer 
review system (Garfield 1989; Martin and Irvine 1981, 1983). By the early 1960s 
changes within the scientific community required greater funding selectivity. Big (and 
increasingly Expensive) Science continued to grow, with many new fields emerging. 
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creating in turn more opportunities and demands for funding. Research became more 
multidisciplinary and collaborative, spawning extensive projects requiring increased 
coordination. This period also saw the rise of "strategic research," as well as rising 
economic constraints requiring increasingly difficult funding choices be made between 
competing research proposals (Garfield 1989; Martin and Irvine 1983; Martin 1996). 

Peer Review System 

Traditionally scientific evaluation was conducted by the peer review system 
(Garfield 1989; van Raan, 1996). However, this system has come under increasing strain 
and criticism for a number of reasons. For instance, it is increasingly difficult to get 
impartial reviewers in an environment of growing concentration of research facilities into 
fewer units (Garfield 1989; Martin 1996) and increased competition and rivalry for 
recognition and funding (Martin and Irvine 1983; van Raan 1996; Chubin and Hackett 
1990). The peer review system is increasingly perceived as an "old boy network" that 
favors established fields, and the protection of colleagues’ declining research areas over 
emerging new ones (Garfield 1989; van Raan 1996; Chubin and Hackett 1990). There is 
also increasing recognition of the "halo effect" or the tendency to fund higher status and 
more visible programs (Garfield 1989) by an over-reliance on institutional or individual 
reputation (Martin and Irvine 1981, 1983; Martin 1996), or by an unwarranted extension 
of a university’s overall prestige to all of its constituent departments (Koenig 1983). 
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Others complain of inconsistent reviewer assessment criteria and interpretation 
(Garfield 1989; Seglen 1997) based in part on a tendency of reviewers to interpret the 
research efforts of others through their own research activities and interests (Martin and 
Irvine 1983). Such inconsistent performance is made worse by a growing lack of 
consensus about the nature of doing "good science", who its practitioners are, and where 
the "best" lines of inquiry are, especially in relation to newer research areas (Garfield 
1989; Seglen 1997; Chubin and Hackett 1990). Part of the problem is due to an 
underlying tendency of most research scientists to conform to conventional and accepted 
patterns of belief (Martin and Irvine 1981, 1983). In addition, little thought is given to 
the actual cost of review process in terms of administrative support and scientists' time, a 
factor often ignored or minimized (Garfield 1989). Also overlooked is the reality that 
most if not all reviewers cannot “guarantee the originality and accuracy of the 
manuscripts they review” because they have neither the time nor the resources nor in 
some cases the expertise to replicate or even substantially recalculate or reanalyze the 
data (Chubin and Hackett 1993, 87). Hence the end-product of peer review is ultimately 
the result of expert perceptions formed by a complex interaction of social and intellectual 
processes often not related to issues of quality (Martin & Irvine 1983; Chubin and 
Hackett 1990). 
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Bibliometric Indicators 

For all its faults, the peer review process is, and probably will remain, an essential 
part of the assessment of quality in science (Garfield 1989; Martin and Irvine 1983; van 
Raan 1996; Rinia et al. 1998). Improvements to the system are needed, however, and 
among those su gg ested is the use of quantitative, objective scientific indicators to 
compliment the qualitative peer review process (van Raan 1996; Martin 1996). Of the 
various indicators available, the mostly widely used are those derived from the 
bibliometric study of publication counts and citations (Garfield 1989; Martin and Irvine 
1983). 

Publication Counts 

The analysis of publication counts measures the total volume of research output in 
terms of journal articles, which is the accepted medium of reporting the results of 
scientific research by scientists (Garfield 1989; Martin and Irvine 1983; Martin 1996). 

The assumption is that each published scientific paper contributes to the advancement of 
scientific knowledge (Martin and Irvine 1983). There is also some empirical evidence 
that publication counts of individuals tend to correlate to a reasonable degree with other 
measures of scientific achievement, such as peer ranking and funding (Garfield 1989). At 
the level of individual assessment, publication counts can be as useful and perhaps more 
accurate than citation counts (Phelan 1999). 
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The use of publication counts is not without its critics. Martin and Irvine (1983) 
and Seglen (1992) pointed out that each individual paper does not necessarily make an 
equal contribution to science, as only about 1 5% of the scientists publish approximately 
50% of the papers (Seglen 1992). Publication counts provide no indication of the quality 
of research output (Garfield 1989; Bayer and Folger 1966). Studies by Rinia et al. (1998) 
and Martin and Irvine (1983) found no evidence that publication counts of institutions 
and research groups correlate with either peer review ratings or other citation-based 
bibliometric indicators (see also van Raan and van Leeuwen 1998; Noyons, Moed, and 
Luwel 1 999). The use of publication counts also ignores the importance in scientific 
communication of other, non-journal publications, both formal (e.g., technical reports) 
and informal (e.g., pre-prints) (Garfield 1989; Seglen 1997; Moravcsik 1973). 

Publication practices also vary among journals and research areas, institutions, and 
countries (Garfield 1989). A reliance of publication counts to measure research 
productivity may possibly encourage undesirable publication practices by authors, such 
as gratuitously conferring co-authorship, and the unnecessary division of a paper into 
"Least Publishable Units" (Garfield 1989; Schoonbaert and Roelants 1996). It also begs 
the question of how to assign publication credit to multiple authors of single papers in an 
environment that increasingly produces only multi-authored papers (Schoonbaert and 
Roelants 1996). 
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Citation Analysis 

The reliance on publication counts alone brings with it the problem of quantity vs. 
quality (Bayer and Folger 1966; Hagstrom 1971). Publication counts over emphasizes 
the relative importance of the quantity of publications while saying nothing about the 
publication’s “intrinsic quality and external impact” (Schoonbaert and Roelants 1966, 
740; Martin and Irvine 1983). One solution is to use a system of weighting to account for 
the relative merit of each publication. Unfortunately, these weighting systems are usually 
devised in a very subject manner without “adequate theoretical basis for the choice of 
weights” (Martin and Irvine 1983). The use of citation analysis to provide a measure of 
quality (i.e., impact or influence) for evaluating scientific publications avoids this 
problem by using readily available, empirical data on the number of citations received by 
a given publication. It assumes that a publication’s impact is on subsequent publications, 
with each instance of such influence showing up as a reference in the influenced (or 
impacted) paper (Martin and Irvine 1983). 

Background 

The use of citation analysis evolved out of a concern that publication counts alone 
did not account for another, perhaps more important, aspect of the scientific enterprise, 
the quality (or utility or the usefulness) of the research (Bayer and Folger 1996; Hagstrom 
1971; Martin and Irvine 1983). 
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The origins of citation analysis, for all practical purposes, is tied to the 
development of the citation indexing database created by the Institute for Scientific 
Information (ISI), consisting of the reference lists extracted from a large number of 
scholarly journals (Seglen 1997). Citation analysis began in earnest with ISI's 
publication in 1961 of the Science Citation Index (SCI) (MacRoberts and MacRoberts 
1989). It was designed as literature research tool, organized to show each reference, 
listed alphabetically by the first author's last name, the number of times it was cited in the 
previous year, and by whom (Seglen 1997). Over the years, the SCI was retrospectively 
expanded to 1945, and ISI added quarterly updates, five-year accumulations, the Social 
Science Citation Index (SSCI), the Arts and Humanities Citation Index (AHCI), CD- 
ROM (Schoonbaert and Roelants 1996) and on-line versions {Web of Science) covering 
1970-present (Web of Science 2000). The total coverage by 1990 was 18,000,000 source 
publications with over 217,000,000 citations (Schoonbaert and Roelants 1996), with an 
annual processing rate by 1998 of approximately 12,000,000 references for 4500 journals 
(Institute for Scientific Information, 1999; Schoonbaert and Roelants 1996). 

Assumptions 

Citation analysis assumes that "an intellectual link exists between the citing 
source and reference article" (Garfield 1989, 96; Smith 1981), and that cited references 
are a record of their influence on an author's work. Such a view is based on a normative 
theory of science developed by Merton in 1957 (as cited in MacRoberts and MacRoberts 
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1989) in which "bibliographies are lists of influences,” and scientists will "cite the work 
that they have found useful in pursuing their own research" (MacRoberts and 
MacRoberts 1989, 342). According to Merton's 1968 study (as cited in Case and 
Higgins 2000), citations are part of the rewards system in science, and come in the "form 
of social recognition, even when critical in nature" (p. 635). Citation analysis also 
assumes that a failure to cite is a rarity, and that most papers cite properly and accurately 
(MacRoberts and MacRoberts 1989; Smith 1981). 

It is also assumed that citations can function as an indicator of scientific quality 
(Seglen 1997; Smith 1981) and importance, and the value of a publication are related to 
its explicit use as a citation in later papers (Schoonbaert and Roelants 1996). McAllister, 
Anderson, and Narin (1980) reported correlations of 0.5 to 0.8 between high citation 
counts and other recognized quality measures such as honorific awards like the Noble 
Prize (Garfield 1989) and the National Medal of Science (Myers 1970), academic 
position, and membership in scientific academies (Myers 1970; Phelan 1999). Citation 
frequencies of researchers have also been found to correspond to the overall peer ratings 
of departments or institutes to which they belong (Myers 1970; Garfield 1989). 

For these reasons, the results of citation analysis have become highly influential 
within the scientific community. It has had an increasing effect on scientific reputations 
as well as individual bids for tenure, promotion, and research funding (Schoonbaert and 
Roelants 1996). However, citation analysis, like publication counts, is not without its 
critics, and many of the criticisms of publication counts apply to citation analysis as well. 
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Criticisms 

The criticisms of citation analysis fall into five general categories; citation 
motivation and practice, citation database limitations, field dependent factors, consumer 
perceptions, and journal impact factors. Although each category has generated quite a bit 
of literature in its own right, especially journal impact factors, this paper will briefly 
address only the main issues within each category. 

Citation motivation and practice 

Citation motivation is more complex that just giving intellectual credit 
(MacRoberts and MacRoberts 1989; Case and Higgins 2000). Citation motivation can be 
idiosyncratic, driven by self-interest such as "hat-tipping" (paying homage) and attempts 
to curry favor with mentors, editors, or grant agencies (Case and Higgins 2000; Garfield 
1989; Smith 1981). 

Citations can be perfunctory as well, as was demonstrated by Moravcsik and 
Murugesan’s (1975) study of a random sample of high energy physics papers publishing 
in the journal Physical Review. They found that slightly more than 40% of the references 
were to acknowledge papers that had done work in the same general research area, and 
not necessarily due to their contribution to the advancement of science (Moravcsik and 
Murugesan 1975). They note that such a high percentage of perfunctory citations raises 
“serious doubts about the use of citations as a quality measure” (Moravcsik and 
Murugesan 1975, 90). 
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Seglen (1997) found that among practicing scientists, the "primary criterion for 
reference selection" is the "utility within research rather than pure scientific quality" (p. 
1050). Smith (1981) however noted that not all items used are necessarily cited, and “not 
all items cited were used” (p. 87). She also observed that citations do not always reflect 
the relative merit (“quality, significance, impact”) of the cited item, but other factors as 
well, such as ease of accessibility in terms of its form, origins, language, and age (Smith 
1981, 87-88). 

Various citation practices can cause problems as well. For example, incorrect 
work can be highly cited (Garfield 1989; Martin 1996) as evidenced by Moravcsik and 
Murugesan’s (1975) finding that 13-16% of their sample’s references were negational, 
disputing the correctness of the reference. Authors can also create erroneous citations by 
depending on secondary sources rather than reading the original works (Case and Higgins 
2000). Such papers can make a positive impact however by stimulating further research 
to confirm or deny the conclusions found in the erroneous papers (Martin 1996). 
Moravcsik and Murugesan (1975) noted that “a paper can be wrong on several levels,” 
not just incorrect when written. They also pointed out that “all scientific papers turn out 
to be ‘wrong’ eventually in the sense that they are replaced by something better: (p. 92; 
see also Moravcsik 1973). Usually though, in the sciences erroneous papers are simply 
ignored (Garfield 1989). 

Certain works, such as methodological papers and review articles, tend to be 
disproportionately used and cited (MacRoberts and MacRoberts 1989; Garfield 1989; 
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Martin 1996) due in part to lack of journal space preventing the citation of all the sources 
drawn upon by a researcher (Seglen 1997). There is no reason however to consider this 
type of article inferior or of less importance than other types of articles. Though they do 
not advance knowledge directly, methodological papers and review articles do so 
indirectly by providing “a foundation upon which other scientists can advance 
knowledge” (Phelan 1999, 123). 

Self-citation, which can run from 10-30% (MacRoberts and MacRoberts 1989; 
Seglen 1997), may artificially inflate the citation rate (Garfield 1989), and lead to a form 
of cronyism where authors cite their own friends and colleagues thereby reducing the 
value of citation analysis (Phelan 1999; Martin 1996). There is some empirical evidence 
to the contrary, however. In their study of cancer research Lawani and Bayer (1983) 
observed that their results remained unchanged “whether or not self-citations are included 
(p<0.001)” (p. 64). The argument against the inclusion of self-citations also ignores the 
fact that the more papers a researcher has published in a specific area, “the larger the 
reservoir grows for justifiable self-citation" (Schoonbaert and Roelants 1996, 744). 

Phelan (1999) finds that due to the cumulative nature of focused individual research, self- 
citation is “not only a natural and acceptable procedure, it is also a useful and informative 
one” (p. 123). It is only a potential problem when evaluating individual researchers. 
Otherwise self-citation is probably not an important problem at more aggregated levels of 
study (Phelan 1999). 
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As for the related problem of ‘cronyism,’ Phelan (1999) thinks that fact that 
“academic leaders cite each other extensively, however, simply reinforces that fact that a 
large number of citations tends to indicate a scholar’s position in the field’s hierarchy.” It 
is actually a “manifestation of the power relations existing within a field. That citation 
counts reflect this reality is not a methodological shortcoming” (Phelan 1999, 127). 

Martin (1996) found no evidence that cronyism existed, except as a “modem legend” (p. 
357). However, these positions do not address the underlying problem of the influence 
that such scholarly hierarchies can have on peer review processes, which are instmmental 
in determining research direction, grant funding, as well as who gets published in which 
journal (Daniel 1993; Chubin and Hackett 1990). 

The focus on citations to formally published papers ignores the informal 
influences on an author's work, such as conference discussions, pre-prints, and in-house 
shop talk (MacRoberts and MacRoberts 1989). In addition, there can be an "obliteration 
phenomena" at work, where theories and techniques become so assimilated into current 
scientific knowledge that their originators cease to be credited (Garfield 1989; Moravcsik 
1973; Smith 1981). 

Citation database limitations 

The criticisms concerning the database limitations are particularly important 
considering that the ISI citation databases are the only source for citation data (Garfield 
1989). The main problems concern information retrieval and the database’s publication 
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coverage. Retrieval accuracy can be affected by the presence of author homographs 
(different authors having the same name), inconsistent use of initials, inconsistent 
spelling of foreign names, and citing errors (Garfield 1989; MacRoberts and MacRoberts 
1989; Seglen 1997). The homograph problem can be a particularly vexing when dealing 
with inconsistently anglicized versions of East Asian names. These problems of author 
identification (misspelled names, homographs, and institutional addresses) are not 
always, in themselves, intractable. The solutions, while tedious and time consuming, are 
not impossible. They do however require the examination of every paper and 
institutional affiliation of every author being considered, as well as accepting that ‘data 
cleaning’ cannot be done exclusively by computer (Phelan 1999). 

As for the database’s publication coverage, it only covers the journal literature 
(van Raan 1996). The coverage also changes over time at an estimated annual rate of 
approximately 10% (Schubert, Glanzel, and Braun 1989) as some Journals are dropped 
and others added, and the expansion of publication types in 1977 to include conference 
proceedings (Garfield 1989; MacRoberts and MacRoberts 1989; Seglen 1997). In all 
fairness though, it is simply not feasible for the ISI database coverage to be completely 
comprehensive (Schoonbaert and Roelants 1996). Space and economics preclude the 
inclusion of hundreds of low impact Journals into the citation database (Garfield 1996). 
Also the lack of coverage may not be as serious a problem as the critics maintain. For 
example, the Australians found strong relationships between publication in all Journals 
and publication in ISI covered Journals at both the university and departmental levels of 
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comparison, especially in the natural sciences (Phelan 1999). Nederhof, Meijer, Moed, 
and van Raan (1993) discovered in their study of a Dutch agricultural university that ISI 
covered journals accounted for over 80% of the citations in the natural and biosciences 
departments. Therefore, Phelan (1999) thinks that for the vast majority of research fields, 
the ISI data is “likely to provide an excellent indication of total research activity” (p. 126; 
see also Korevaar and Moed 1996). 

The coverage of the citation database is also unequal, varying between research 
fields, with some being better represented than others, e.g., 90% for chemistry vs. 30% 
for biology (Seglen 1997; Schoonbaert and Roelants 1996). However it is important to 
note that the fluctuating coverage reflects ISI’s attempts to select and maintain database 
coverage of the top 10% of the journals that provide 90% of the significant literature. To 
do so requires a periodic adjustment to the number of journals covered (Schoonbaert and 
Roelants 1996) to reflect the changes in science over time (Phelan 1999). By adding 
journals as they begin to attract substantial citations, ISI is actually creating a current and 
dynamic journal set that more accurately reflects contemporary academic interests, while 
also making it more suitable for use in studies focused on a single point in time as well as 
for those examining change over time. This means, for example, that an institution with a 
higher ranking in a dynamic journal set indicates that it is receiving more recognition in 
research areas that are currently important (Phelan 1999). It may also be equally 
reflective of the state of contemporary of scientific politics as much as the objective, 
rational evaluation of scientific research (Chubin and Hackett 1990; Daniel 1993). 
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Other critics point out that the database also has a bias in favor of journals using 
the English language and the Roman alphabet (Garfield 1989; MacRoberts and 
MacRoberts 1989; Seglen 1997). However, practically speaking, this is only a problem 
in studies examining highly specialized fields or that are assessing the importance of 
regionally specific research in non-English speaking countries (Phelan 1999). 

Field dependent factors 

There are. inconsistencies within the ISI citation database caused by field 
dependent factors or factors relating to a specific discipline or research area. Of 
particular concern are the varying citation (and publication) practices between research 
fields and over time (Moravcsik 1973; Garfield 1989; MacRoberts and MacRoberts 
1989). The size of the research field can potentially effect the probability of being cited, 
with larger or broader research fields being more likely to be cited than smaller or 
narrower ones (Garfield 1989; Seglen 1997). The rate at which papers become obsolete 
varies with field, with lower cited papers become obsolete quicker than higher cited ones 
(Garfield 1989). It can also be difficult to place a newer, multi-disciplinary research area 
into one of the current disciplinary or research-based publication groups (Garfield 1989). 
There is also the problem of citation lag, which varies between research fields. Some 
critics see this variation in citation lags as disproportionately rewarding the faster, 
“hotter” research fields with faster citation peaks, often measured in months and 
penalizing fields with a longer, slower citation peaks measured in years (Schoonbaert and 
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Roelants 1996). For these reasons alone, the results of citation analysis cannot be 
compared across different research fields (Moravcsik 1973; Seglen 1997; Martin 1996). 

Consumer perceptions 

For researchers familiar with the strengths and weakness of citation analysis, the 
evaluation of research quality means that citations measure ‘impact’ rather than ‘quality’ 
(cf van Raan 1996). Impact, when defined as the actual influence of a paper on the 
surrounding research activity during a specified period of time, is easier to operationalize 
in a bibliometric study than is ‘quality’, a more elusive and difficult to define concept 
(Phelan 1999; Martin and Irvine 1983; Martin 1996; Garfield 1989; Korevaar and Moed 
1996; Smith 1981). 

Yet consumers of bibliometric studies, such as university administrators and 
government officials, are more than willing to make a strong conceptual link between 
citation analysis and the assessment of research quality, which reflects their primary 
interest in bibliometric analysis. The problem revolves around the different 
understandings or perceptions of indicator precision held by bibliometric researchers and 
consumers. Consumers tend to have greater confidence in the precision of the 
bibliometric indicators than do researchers, reflecting their focus on broad results rather 
than on the details of measurement, which is of primary interest of bibliometric 
researchers (Phelan 1999). Phelan (1999) recommends that given the complexities of 
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measurement and the emotionally charged label ‘research quality,’ it would be better to 
emphasize to consumers the concept of visibility or impact rather than quality. 

Journal impact factors 

One of the most controversial aspects of citation analysis was (and continues to 
be) the creation and use of ISI’s journal impact factor (for more detail, see Appendix A). 
Journal impact factors (JIFs) were originally created as indicators of journal citedness, 
and as a means of comparing journals of differing sizes (Garfield 1996). JIFs were 
designed to neutralize the advantage of larger journals with their greater pool of 
potentially citable articles over smaller ones (Schoonbaert and Roelants 1996; Institute 
for Scientific Information 1999). They are defined as “ratios obtained from dividing 
citations received in one year by papers published in the two previous years. Thus, the 
1995 impact factor counts the citations in 1995 journal issues to “items” [i.e., “original 
research and review articles, as well as notes”] published in 1993 and 1994” (Garfield 
1996, 1; Institute for Scientific Information 1999, 14). If the coverage were for one 
year, it would be biased in favor of rapidly changing fields, whereas coverage of greater 
than two years would make the JIFs less current (Garfield 1999). The results are 
published annually in ISVs Journal Citation Reports 1997; Schoonbaert and 

Roelants 1996; Institute for Scientific Information 1999). 

While originally intended as a way to compare journals, the use of the JIF has 
evolved, especially in Europe, to include the assessment of the quality or impact of not 
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only journals, but also research groups and individual authors (Seglen 1997; Garfield 
1999). This change in use has caused problems, especially in the evaluation of individual 
author impact because the journal comparison involves a relatively larger population of 
articles and citations, far more than the vast majority of authors usually produce (Garfield 
1999). 

A particularly controversial use of the JIF is as a surrogate measure or substitute 
for actual citation counts. This use of the JIF is based on the assumption that journal is 
representative of its articles (Seglen 1997). Using the JIF as a surrogate is a relatively 
simple and inexpensive alternative to citation counts, which could be especially useful for 
large corporations and universities working with huge datasets (Schoonbaert and 
Roelants 1996). However, the use of JIFs as citation surrogates would only work if the 
frequency of article citation rates formed a Gaussian (normal) distribution around the 
population mean (i.e., the journal impact factor). Actual journal article citedness is very 
skewed with only about 15% of the articles accounting for approximately 50% of the 
citations (Seglen 1997; Schoonbaert and Roelants 1996; Phelan 1999). In addition, 
Nederhof et al.’s (1993) study found the JIF to be a poorer predictor of actual impact than 
the use of actual impact data. 

The use of the JIF as a surrogate, however, can say something about the relative 
value of an individual paper accepted by a high impact journal for publication. This 

S 

acceptance by the journal’s peer reviewers and editorial board can be seen as an 
acknowledgement of the paper’s worth and accreditation (Schoonbaert and Roelants 
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1996). This position has some empirical support from McAllister, Anderson, and Narin’s 
(1980) study of scientific journals, which found correlations between a journal’s impact 
factor and its peer assessment ranging from 0.4 to 0.9. Also, absolute citation counts 
ignore the greater weight that citations in higher prestige journals should get (Garfield 
1989; cf Davis and Royle 1996). Although Garfield (1996) warns against the use of the 
JIF in place of actual article citation counts as a dangerous expedient, especially in 
faculty evaluation, he acknowledges that JIFs could be used as surrogates in unusual 
circumstances. 

There are other criticisms of the journal impact factor. For instance, the way that 
JIFs are determined could possibly cause inflated results. This is due to the inclusion of 
citations from all types of documents in the numerator, while only journal articles, 
technical notes, and reviews used in the denominator (Seglen 1997; Institute for 
Scientific Information 1999). However, the addition of all documents types, most of 
which were not intended to be cited, would unfairly lower the JIF of journals that publish 
mainly currently citable source documents, such as The Lancet and the New England 
Journal of Medicine (Schoonbaert and Roelants 1996). In addition, several studies have 
shown the current source document types in the SCI account for 69-95% of the 
publications and their citations (Moed and van Leeuwen 1995; Bourke and Butler 1993). 

Another criticism is that the JIF, based on citations up to two years after 
publication, penalizes journals with longer publication lags, and those in slower moving 
research fields in which maximum potential citation peaks are not reached until three to 
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five years after publication (Seglen 1997). It may be however that this problem with the 
two year ‘citation window’ is not as significant as it seems. Garfield (1999) found that 
when journals within the same disciplinary categories are examined, their impact 
rankings based on one year, seven year or fifteen year JIFs do not differ significantly. 

There are also some fears that authors will shun journals with lower JIFs in favor 
of those with higher ones regardless of actual distinctions between quality, scientific 
suitability, and editorial fairness (Seglen 1997). However, Garfield (1996) points out that 
the JIF was never intended as a substitution for the editorial judgment necessary to select 
the best papers. In the final analysis, says Garfield (1996), “impact simply reflects the 
ability of journals and editors to attract the best papers available” (p. 1). 

Conclusions about citation analysis 

No system of counting result or ranking can be made fool proof. "In the absence 
of other reliable methods based on purely quantitative methods, this bibliometric 
approach is arguably the next best thing to the direct assessment of the intrinsic quality of 
the publication" (Schoonbaert and Roelants 1996, 740). 

In general, most of the criticism of citation analysis does not reject it for a lack of 
objectivity, but rather because it does not take into account enough of the particular 
context of individual cases. Unfortunately, excessive concern with contextual details of 
individual cases can lead to the continuous creation of refining parameters, each 
calibrated by discipline-specific coefficients (e.g., weighted or relative JIFs) to correct 
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discipline-related inequities, as well as the inclusion of things like the 'fractional citation 
counting' (giving each author of a single paper a fraction of the credit). Such an approach 
would soon make any ranking formula unworkable (Schoonbaert and Roelants 1996). 

Also, one should keep in mind that much of the criticism of the iSl’s citation 
indexes and its Journal Citation Reports appear to be based on relatively limited bodies 
of evidence. The ISI has always acknowledged their limits, and warned against the 
indiscriminate use of their products, especially for individual judgments (Schoonbaert 
and Roelants 1996; Institute for Scientific Information 2000). 

Bibliometric methods should never be accepted as autonomously conclusive 
(Schoonbaert and Roelants 1996). The use of multiple bibliometric indicators in 
conjunction with other imperfect indicators, such as the results of peer reviews (Garfield 
1989; van Raan 1996; Martin 1996; Phelan 1999), can reduce much of the negative 
emotional response to citation analysis (Schoonbaert and Roelants 1996; Martin 1996), as 
well as improve the overall accuracy of research assessment (Martin and Irvine 1983; 
Martin 1996). Several important limitations of citation analysis to remember are that 
citation patterns vary between disciplines, not all specific research topics are equally 
popular at any given time, and the presence of English language bias in the citation data 
(Schoonbaert and Roelants 1996). 

It is also important to remember that bibliometric indicators, though reasonably 
objective in their creation, still require some degree of interpretation of their meaning, 
leaving them open to being manipulated into supporting existing power structures by 
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maintaining or bolstering their influence over research, funding, journal editorship, and 
publishing (Daniels 1993; Chubin and Hackett 1990). In addition, bibliometric indicators 
appear to work best when examining entities with large numbers of citations, which 
means that the examination of individual cases needs to be done with the greatest care 
(Phelan 1999). 



Bibliometric Studies of Academic Science Departments 
Availability of Published Studies 

Compared to the published literature analyzing the method and theory of citation 
analysis, the number of studies applying citation and publication based bibliometric 
indicators is relatively meager. Of those available, only nine were found that evaluated 
departmental level (or the equivalent) research productivity. Only one published 
bibliometric study evaluated nutrition and food research, while the other eight examined 
academic science departments in such fields as experimental psychology, mathematics, 
and condensed matter physics. All nine studies were European in origin, with six 
produced by a group of researchers in the Netherlands associated with the Centre for 
Science and Technology Studies (CWTS) (see Table 1). None of the references 
consulted for this study gave any indications why there are so few published 
departmental level bibliometric studies available. Part of the answer may be in the desire 
of the studied departments to keep the results confidential (e.g., see Van Leeuwen, Rinia, 
and van Raan 1998). 
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Table 1 . Nine bibliometric studies of academic science departments, research groups, and 
institutes, by type of research unit, location, and study source. 



Type of Research Unit 


Location 


Source of Study 


Agricultural university 
departments 


Netherlands 


CWTS (Nederhof, Meijer, 
Moed, and Raan 1993) 


Experimental psychology 
departments 

High energy physics: electron 
accelerator research centers 


United Kingdom 
United States 
Netherlands 

United Kingdom 


CWTS (Nederhof and 
Noyons 1992) 

SPRU (Martin and Irvine 
1981) 


Mathematics departments 


Greece 

Belgium 


BIU (Zachos 1991) 

CWTS (Korevaar and Moed 
1996) 


Micro-electronic research 
center 


Belgium 


CWTS & AWI (Noyons, 
Moed, and Luwel 1999) 


National condensed matter 
physics programs 


Netherlands 


CWTS (Rinia, van Leeuwen, 
van Vuren, and van Raan 
1998) 


Nutrition and food research 
institute 


Netherlands 


CWTS (van Raan and van 
Leeuwen 1998) 


Radio astronomy research 
centers 


United Kingdom 

Germany 

Netherlands 


SPRU (Martin and Irvine 
1983) 



Note(s): 

For purposes of this study, academic departments includes not only those discipline-specific 
organizational units traditionally associated with a particular university, such as the Department 
of Physics at the University of Tennessee, but also research centers, academic institutes, or 
research groups, often multi-disciplinary, affiliated with a specific university or consortium of 
universities such as the Inter-university Centre for Micro-Electronics in Leuven, Belgium and the 
Max-Planck-Institut fur Radioastronomie in Boim, Germany. 
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Table 1. (continued) 

Source of Study. The associated agency responsible for the conducting the study, followed by the 
reference to its published version. 

CWTS: Centre for Science and Technology Studies, University of Leiden, Netherlands. 

A Wl\ Science and Innovation Administration, Belgium. 

B1U\ Bibliographic Information Unit, University of loannina, Greece. 

SPRU: Science Policy Research Unit, University of Sussex, United Kingdom. 

Sources(s): 

Korevaar, J.C., and H.F. Moed. 1996. Validation of bibliometric indicators in the field of 
mathematics. 5c/'e«ro/wern'« .37, no. 1; 117-130. 

Martin, B.R., and J. Irvine. 1981. Internal criteria for scientific choice; An evaluation of research 
in high-energy physics using electron accelerators. Minerva 19: 408-432. 

Martin, B.R., and J. Irvine. 1983. Assessing basic research. Some partial indicators of scientific 
progress in radio astronomy. Research Policy 12: 61-90. 

Nederhof, A.J., R.F. Meijer, H.F. Moed, and A.F.J. van Raan. 1993. Research performance 
indicators for university departments: A study of an agricultural university. Scientometrics 27: 
157-178. 

Nederhof, A.J., and E.C.M. Noyons. 1992. Assessment of the international standing of university 
departments’ research: A comparison of bibliometric methods. Scientometrics 24: 393-404. 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation analysis for 
evaluative bibliometric purposes: A bibliometric study. Journal of the American Society for 
Information Science 50, no. 2: 115-131. 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. Comparative 
analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of 
condensed matter physics in the Netherlands. Research Policy 27: 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, applied 
research using bibliometric methods in nutrition and food research. <http:// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 

Zachos, G. 1991. Research output evaluation of two university departments in Greece with the 
use of bibliometric indicators. Scientometrics 2\, no.2: 195-221. 
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Brief Comparison of Available Studies 

The early CWTS studies (Nederhof and Noyons 1992; Nederhof et al. 1993) used 
relatively small data sets to focus on several of aspects relating to a department’s research 
performance, such as comparative national and international standing (Nederhof and 
Noyons 1992). These early studies suffered from little or no literature review or analysis, 
generally incomplete reporting, and overall poor presentation. For instance, the Nederhof 
and Noyons (1992) study was so badly presented as to render it virtually valueless. In 
contrast, the later CWTS studies (Korevaar and Moed 1996; Rinia et al. 1998; van Raan 
and van Leeuwen 1998; Noyons, Moed, and Luwel 1999) were a vast improvement, 
using large data sets originally developed for other related projects, better research design 
and execution, as well as much improved presentation. These studies also shared the goal 
of discovering the specific relationship that their bibliometric indicators had with other 
indicators, especially peer review ratings. 

The Zachos (1991) study of two Greek mathematics departments also suffered 
from the lack of a literature review as well. However, it is an excellent example of how 
multiple indicators, bibliometric and otherwise, can be used to provide a more complete 
evaluation of departmental research performance. 

In many respects, the one of the earliest studies in the group, Marin and Irvine’s 
(1983) examination of four radio astronomy research centers, is one of the best, building 
on their previous work (Martin and Irvine 1981). It has a good introduction, an excellent 
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literature review and analysis, and a well-conceived and executed study with generalized 
(and generalizable) results. 

A common theme that unites these various studies is a focus on the nature and 
behavior of a variety of bibliometric indicators rather than an interest in evaluating the 
research output of departments per se. This emphasis tends to make these studies 
exploratory rather than explanatory in nature. These studies also tend to address two 
associated issues rather frequently, the problems of self-citation and allotting publication 
credit to multi-authored papers. 

Evaluation of the Bibliometric Indicators Used 

The nine studies under consideration together generated twenty-six different 
publication and citation-based indicators. The evaluation criteria for determining the 
suitability of any of these indicators for this study involved the clarity and precision of 
description, the ease of calculation, the suitability for application to the circumstances of 
this study, and the interpretation of a particular indicator. The clarity and precision of the 
description of an indicator is required to insure that it is measuring what one thinks it is 
measuring, while the ease of calculation is important due to the time, staffing, and data 
source constraints of this study. The application of an indicator in a study will allow a 
determination of its suitability for the research conditions of this project, while its 
interpretation will indicate what information it can and cannot be expected to deliver. 




46 



34 



Of particular value are those indicators that, in addition to meeting the 
aforementioned criteria, also indicate show a strong correspondence or correlation with 
other methods of evaluating research quality, particularly peer review. For example. Van 
Raan (1993, cited in van Raan and van Leeuwen 1998) considers the combination of 
bibliometric indicators and peer review ratings to represent the best method for 
evaluating research performance. McAllister, Anderson, and Narin’s (1980) study 
reported correlations of 0.7 to 0.9 between departmental publication and citation rates and 
their peer rankings. There are also other studies that report findings that support the 
linkage between citations and peer review ratings such as the Rinia et al. (1998) 
evaluation of Dutch research into condensed matter physics, Koenig’s (1983) study of 
pharmaceutical research performance, Clark’s (1975) study of psychology, and Lawani 
and Bayer’s (1983) study of cancer research. 

Types of bibliometic indicators 

Upon examination, these twenty-six indicators can be grouped into three general 

types, primary indicators, secondary indicators, and incidental indicators, based on their 

/ 

description, method of calculation, application, interpretation, and correlation with other 
non-bibliometric indicators (for more detail concerning the evaluation procedure, see 
Appendix B). A primary indicator is a partial measure of a specific aspect of research 
performance, and empirically correlates with at least one other indicator, such as peer 
review results. A secondary indicator is mainly used as a basis for calculating primary 
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indicators, and on occasion, can be used as a primary indicator in its own right. An 
incidental indicator appears to have no explicitly useful or practical purpose in the study 
in which it appears. Of the twenty-six indicators under evaluation, seven are primary 
indicators, three are secondary indicators, and sixteen are incidental indicators. Since the 
latter seem to have no discernable practical use, only the ten primary and secondary 
indicators will be considered for use in this study (see Table 2). 

Problem of Self-Citations 

One of the problems that all the studies had to resolve is what to do with self- 
citations (or self-cites). Self-citations were handled in a variety of ways by the studies 
examined. For instance, Zachos (1991) separately counted and reported self-citations, in- 
house citations, and “foreign” citations. Van Raan and van Leeuwen (1998), Rinia et al. 
(1998), Noyons, Moed, and Luwel (1999) used both indicators that included self-cites, 
and some that did not. Nederhof et al. (1993) appeared to have left the self-cites in the 
data. 

The crux of the self-citation issue seems to lay in the oft implied notion that the 
primary motive driving self-citation is self-aggrandizement, with the resulting distortion 
of the citation database (MacRoberts and MacRoberts 1989; Seglan 1997). There seems 
to be little empirical evidence however to support this assumption, and some research 
(Case and Higgins 2000) that indicates that an author’s motivation to self-cite does not 
vary appreciably from the motivation to cite others. To remove automatically all self- 
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Table 2. Primary and secondary bibliometric indicators used in nine studies,, by indicator, 
definition, and interpretation. 



Indicator 


Definition 


Interpretation 


Primary 


C 


Total citations. 


General impact measure. 


CPP 


Average citations per 
journal publication. 


Actual impact normalized for size 
of publication output. 


CPP/FCSm 


Citations per paper compared to 
the average citation rate of all 
the journals in a field/sub-field. 


Comparison of actual to expected 
field/ sub-field impact 


CPP/JCSm 


Citations per paper compared to 
the average citation rate of a 
specified journal set. 


Comparison of actual to expected 
journal impact. 


JCSm/FCSm 


Average citation rate of the journal 
set compared to the average citation 
rate of all the journals in the same 
field/sub-field. 


Impact of a journal set on its field. 


HCP 


Highly cited papers (or key papers), 
top 1 % papers that equal or exceed a 
given citation threshold. 


Number of high impact 
“discoveries”produced by a 
research group. 


PPR 


Publications per researcher. 


Research output normalized for 
department size. 


Secondary 


FCSm 


Average citation rate of all the 
journals in a field/sub-field. 


Average field impact. 


JCSm 


Average citation rate of all the 
journals in a specific journal set. 


Average journal impact. 


P 


Total number of publications 


Research or publication output. 
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Table 2. (continued) 

Note(s): 

Impact is a limited, partial measure of research quality in terms of influence on contemporary or 
future research. 

^ Publications refer to original research papers, review papers, and technical notes. 

Source(s): 

Korevaar, J.C., and H.F. Moed. 1996. Validation of bibliometric indicators in the field of 
mathematics. 37, no. 1: 117-130. 

Martin, B.R., and J. Irvine. 1981. Internal criteria for scientific choice; An evaluation of research 
in high-energy physics using electron accelerators. 19: 408-432. 

Martin, B.R., and J. Irvine. 1983. Assessing basic research. Some partial indicators of scientific 
progress in radio astronomy. Research Policy 12: 61-90. 

Nederhof, A.J., R.F. Meijer, H.F. Moed, and A.F.J. van Raan. 1993. Research performance 
indicators for university departments; A study of an agricultural university. Scientometrics 27' 
157-178. 

Nederhof, A.J., and E.C.M. Noyons. 1992. Assessment of the international standing of university 
departments’ research: A comparison of bibliometric methods. Scientometrics 24: 393-404. 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation analysis for 
evaluative bibliometric purposes: A bibliometric study. Journal of the American Society for 
Information Science 50, no. 2; 115-131. 



Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. Comparative 
analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of 
condensed matter physics in the Netherlands. Research Policy 27: 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, applied 
research using bibliometric methods in nutrition and food research. <http:// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 

Zachos, G. 1991. Research output evaluation of two university departments in Greece with the 
use of bibliometric indicators. Scientometrics 21, no.2: 195-221. 
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citations appears to unnecessarily penalize researchers working in new fields or in 
currently unfashionable fields, where there is not a large amount of relevant citable 
literature available. It also seems to unfairly penalize the efforts of veteran researchers 
who have built up a body of related work upon which they, like science in general, can 
then build upon for future research and development (Schoonbaert and Roelants 1996; 
Phelan 1999). While self-citation may be more of an issue at the level of assessing the 
individual researcher for tenure and promotion, it seems an unnecessary concern at the 
department level of aggregation (or higher) (Phelan 1999; see also van Raan 2000). 

Problem of Multi- Authored Papers 

The second problem of concern in these bibliometric studies is how to assign 
publication credit to all the authors of a multi-authored paper. Multi-authored 
publications were handled in a variety of ways by the studies examined. For example, 
Zachos (1991) counted them proportionally, adjusting “publication and citation counts” 
and giving “a fraction to every co-author” (p. 198). The other studies either credited one 
paper to each author, or did not state how the credit was distributed. Though no 
consensus seems to exist on this topic, it appears that the most equitable way to handle 
this situation for the foreseeable future is to assign one paper’s credit to each author. 
Though it risks giving excessive credit to people only tangentially related to the project, it 
has the virtue of not penalizing those participants unfortunate enough not to be first 
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authors, as well as keeping the evaluation process relatively simple by not adding another 
layer of calculation and fractional reporting. 

Conclusions 

As can be see from the previous discussion, there are very few published reports 
of the assessing the research productivity of academic departments using citation-based 
bibliometric indicators, and almost none specifically evaluating a nutrition department. 
This study could help remedy this situation by adding a needed example of using 
bibliometric indicators to evaluate both an academic science department, as well as a 
department in the Nutrition research field. It would also be a non-European example 
potentially useful for comparative study. 

As for which indicators to use, based on a review of the available literature, it 
appears that the most useful indicators for a departmental level bibliometric study will 
one or more of those listed in Table 2. It also seems that one can probably safely ignore 
the self-citation phenomena, which would be of greater concern to evaluations of 
individual research performances. Also, at the risk of over-rewarding some people, the 
assignment of one paper’s publication credit to each author of a multi-authored paper is 
probably in the long run the least complicated and more equitable solution to the multi- 
authorship problem. 
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CHAPTER 3 
METHODOLOGY 

“If we knew what we were doing, it wouldn’t be called research, would it?” 

- Albert Einstein (Institute for Scientific Information 2000; Khalid 2000) 

Overall Approach 

The goal of this study is to develop bibliometric indicators that are theoretically 
justifiable and supported by empirical findings and apply them to the evaluation of 
research of four academic research programs. To that end, this study will use a version of 
Martin & Irvine’s (1981, 1983) methodology of multiple converging partial indicators. 
Although not as widely used as one would think (Martin 1996), it has several distinct 
advantages. First, a multi-indicator approach more accurately reflects the multi-faceted 
nature of most basic scientific research, with each indicator revealing a piece of the 
overall picture (see also Hagstrom 1971). By focusing on bibliometric indicators that 
correlate with other indicators, the results of this study could be meaningfully combined 
in the future with peer ratings (Lawani and Beyer 1983; McAllister, Anderson, and Narin 
1980; Hagstrom 1971), esteem indicators (e.g., medals, prizes, invitations to give 
prestigious lectures), number of PhDs awarded, and other partial measure of scientific 
performance (Koenig 1983; Myers 1970; Garfield 1989; Phelan 1999). 

Secondly, a multi-indicator approach tends to facilitate the acceptance by the 
scientific community of the results of non-peer review measures, which in turn are often 
easier and less time consuming to obtain than the more traditional peer review 
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evaluations (Martin 1996). Finally, the use of multiple indicators “minimizes the risk 
that scientists will in some way ‘play the game’ and manipulate the indicators to their 
advantage” without actually improving their research (Martin 1996: 360). For the 
purposes of this study however, the multi-indicator approach will be limited to the use of 
citation-based bibliometric indicators of research productivity (i.e., publications and 
citations) based on the scientific article, the main medium for communicating nutrition 
research results (van Rann and van Leeuwen 1998). 

Level of Analysis 

For this study, the department will be the level of analysis and data aggregation. 
There are several reasons for choosing the department as the level of analysis. Van Raan 
(1996; van Raan 1993, cited in van Raan and van Leeuwen 1998) shows that in most 
cases, the department seems to be the lowest level of aggregation capable of yielding 
statistically useful bibliometric results. It absorbs a large amount of basic research 
funding (Martin and Irvine 1983) as well as matches the administrative, structural, and 
evaluative imperatives of research universities, making it an important focal point for 
research productivity assessment (van Raan 2000). 

Resources 

In terms of resources, there are three possible options for producing a bibliometric 
study. The first option is to buy the study from ISI, which will produce it to virtually any 
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specification (Institute for Scientific Information 2000). The drawback is both the initial 
expense, plus the ongoing cost of any updates purchased in support of annual reports and 
budget requests. The second option is to buy the data from ISI and conduct the analysis 
in house (Institute for Scientific Information 2000). Though less expensive initially than 
buying the study outright, there are other costs to consider, including the cost of long- 
term storage, routine maintenance, data cleaning (e.g., standardizing addresses, names, 
etc), database design and development, and the cost of buying and processing the data for 
annual updates (Moed and van Leeuwen 1995; van Raan 1996; Bourke and Butler 1993). 
The third option, which will be used in this study, is the Do It Yourself approach, taking 
best advantage of what is readily available or easily obtainable at any major academic 
library. The cost is low, requiring only access to the Web of Science database, copies of 
ISI s Journal Citation Reports, connectivity, computer, and software such as Microsoft 
Office 2000. This option is particularly useful to chronically cash-strapped 
organizations, such as many academic libraries, and special libraries at many national 
laboratories, that cannot afford the first two alternatives. The cost is primarily finding 
someone in the organization with the time, expertise, knowledge of the source institutions 
and the people involved, as well as a willingness to do it. 
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Research Design 
Measurement Specifications 
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“Not everything that can be counted counts, and not everything that counts can be 
counted.” - Albert Einstein, from a sign hanging in his Princeton office 
(Institute for Scientific Information 2000; Khalid 2000) 

Assumptions 

Since there is an apparent lack of consensus or empirical evidence in the 
published literature about how to handle self-citations, credit for multi-authored papers, 
and the proper length of citation windows, for the purposes of this study certain 
assumptions will be made concerning self-citations, multi-author publication credit, and 
citation windows. Unlike many other studies (e.g., Noyons, Moed, and Luwel 1999; 
Zachos 1991), there will be no attempt to remove any self-citations from the citation data. 
For a multi-authored paper, each author will be given publication credit for authoring one 
paper. A two-year citation window will be assumed, allowing the use of iSl’s impact 
factors as found in the Journal Citation Report (JCR). The two-year citation window 
used by ISI has the added virtue of simplifying future calculations by utilizing 
sufficiently accurate {pace Moed and van Leeuwen 1995) and readily available data. 

Measurement objectives, types, and categories 

In general, this study is attempting to measure two things; research output and 
research impact. Research output is the amount of journal article productivity, while 
research impact is the influence these papers have on contemporary and future scientific 
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research. Bibliometric indicators are often used to measure research output and impact, 
and can be thought of as being specific instances or examples of four general types of 
measures: absolute, relative, comparative, and benchmark (see Table 3). 

Each type of measure possesses a certain amount of meaning or understanding 
about the item being measured. The degree of potential explanatory power tends to 
increase from the absolute measures with the lowest explanatory potential, to the relative 
measures with greater explanatory potential, and finally to the comparative measures with 
the highest explanatory potential. Benchmark measures derive meaning or explanatory 
power from being used in conjunction with the other three measures. (For an example of 
the relationship between measurement objectives, measurement types, and indicator types 
using the bibliometric indicators from this study, see Table 3.) 

Absolute measures are total counts of like items, such as journal publications or 
citations. They lack any integral context or framework for potentially providing 
meaning, and by themselves offer little or no explanatory power (van Raan 2000). 

Relative measures result from the comparison of absolute measures at the same level of 
aggregation. They contain a limited context that allows the assessment of meaning for a 
specific study group, but little or no explanatory power beyond it. Relative measures can 
also include absolute measures that were converted to a percentage of a study group 
population. Comparative measures are relative measures compared to a benchmark or 
standard value or measure. They possess an integral context extending beyond the 
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- Table 3. Bibliometric indicators selected for use in this study, classified by measurement 
objective, indicator type, and measurement type. 



Measurement 

Objective 


Indicator 

Type 


Measurement 

Type 


Bibliomeric Indicator 


Publication output 
or productivity 


Secondary 


Absolute 


P (Total publications) 


Publication impact 
or influence 


Primary 


Absolute 

Relative 


C (Total actual impact) 

CPP (Average impact per 
publication) 






Comparative 


CPP/FCSm (Average impact on 
research field(s)) 








CPP/JCSm (Average impact on the 
journal set) 








JSCm/FCSm (Average journal set 
impact on the research field(s)) 




Secondary 


Benchmark 


FCSm (Average impact of the 
research field(s)) 








JCSm (Average impact of the journal 
set) 



Note(s): 

Publications refer to original research papers, review papers, and technical notes. 
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originating group to a larger population, and provide a framework for assessing meaning 
in relation to a higher level of aggregation. Comparative measures can also include any 
absolute or relative measures that were converted to a percentage of a higher level of 
aggregation than the study group. Benchmark or standard measures are the average or 
mean values of an aspect of a larger population against which absolute and relative values 
are compared (van Raan 2000). They provide a context that, when joined with an 
absolute or relative measure, converting those measures into comparative measures 
possessing the greatest potential explanatory power. 

To assess the meaning of the bibliometric indicators from the literature that are 
also comparative measures often involves their evaluation in relation to an average 
baseline result, or expected value, usually set at 1.0 (e.g., van Leeuwen and Tijssen 1993; 
Rinia et al. 1998; van Raan and van Leeuwen 1998). Though a use&l way of indicating 
general, relative comparisons (e.g., below average vs. average vs. above average), it 
should be noted that this evaluative method is too restrictive in terms of creating meaning 
or providing potential interpretive value. For example, how much more potentially 
signifrcant or meaningful is a bibliometric indicator with a value of 1. 10 to one with a 
value of 1.45, or 2.01? They are all above 1.0, and therefore are by definition above 
average results. Therefore, the results of such comparisons should be treated with some 
caution as there appears to be little or nothing in the literature about the typical range of 
such indices, which would help shed some light on their relative significance or meaning. 
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Selection of bibliometric indicators 

Eight bibliometric indicators will be used in this study, five primary and three 
secondary indicators (see Table 3). Only two of the primary indicators (C and CPP) and 
one of the secondary indicators (P) from the literature will be used unaltered. Three other 
primary indicators (CPP/FCSm, CPP/JCSm, and JCSm/FCSm) and two secondary 
indicators (FCSm and JCSm) will be used in a modified form, while two primary 
indicators (HCP and PPR) from the literature review will not be used. 

The reasons that the secondary indicators FCSm and JCSm needed to be modified 
for use in this study are found in how and why they were created. Both FCSm and JSCm 
were benchmark measures created to avoid using ISPs Journal Impact Factor (JIF) 
(Korevaar and Moed 1996; van Raan 1996; Nederhof and Noyons 1992). The main 
difference between ISPs JIF and the FCSm and JCSm indicators is the use of a longer 
citation window of three to five years in the calculation of the latter two indicators rather 
than the two-year window used in the calculation of ISPs JIFs, which was considered too 
short to capture the maximum number of citations for a given publication set (Rinia et al. 
1998; van Raan and van Leeuwen 1998; Korevaar and Moed 1996). Since this study is 
using the ISI impact factors, FCSm and JCSm, as well as the three indicators dependent 
on their values, CPP/FCSm, CPP/JCSm, and JCSm/FCSm, were calculated to reflect the 
two-year citation window used by ISI. 
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The remaining two primary indicators from the literature HCP (Highly Cited 
Papers) and PPR (Papers Per Researcher) were excluded from use in this study for 
several reasons. In the case of the HCP indicator, it was due to the lack of sufficient 
evidence for its generalizability beyond the two studies in which it was used (Martin and 
Irvine, 1981, 1983). The link between the HCP indicator and the presence of key papers 
in a discipline is rather weak, and the citation thresholds appear to be excessively 
dependent on a specific data set. The PPR indicator, though a useful way to normalize 
total output by the number of the researchers in a department, suffers from the difficulty 
of securing accurate “counts of staff members who are expected to produce [research] 
results” (Phelan 1999: 133). Since these staff counts can vary year to year, the time 
required to gather the information for this study from four different nutrition departments 
for a five-year period was considered prohibitive. 

Derivation of each selected bibliometric indicator 
Indicator P 

Definition. For the purposes of this study, P is the total number of scientific, scholarly 
publications (articles, notes, and reviews) published by a given Nutrition program in any 
of the scholarly, scientific print journals covered by the ISl’s Web of Science citation 
database and listed in its JCRs over a given period of time (van Raan 2000). 

Calculation. For the purposes of this study, 

P = Ip 
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where Zp is the is the total number of all the publications (articles, notes, and reviews) 
published in a given year by any researcher(s) officially affiliated with one of the 
Nutrition programs in the study. For example, the P published by the University of 
Florida’s (UFL’s) Food Science and Nutrition Program in 1994 is 40 (see Table 7, p. 71). 
Data Source(s). For the purposes of this study, the publication data used to calculate P is 
drawn from the SCl-E section of ISPs Weh of Science citation database. The data is 
drawn for each year of the study 1992-1996, and for each Nutrition program used in this 
study. 

Interpretation. For the purposes of this study, P is considered to be an indicator of the 
scientific productivity (van Raan and van Leeuwen 1998), or research output (van Raan 
2000) of each Nutrition program in terms of print journal publications. 

Indicator C 

Definition. For the purposes of this study, C is the total number of citations (including 
self-citations) received over a two year period (including the year of publication) for all 
of a given Nutrition program’s scholarly, scientific publications (P) in any of the 
scholarly, scientific print journals covered by the ISPs Web of Science citation database 
and listed in its JCRs (van Raan and van Leeuwen 1998; van Raan 2000). 

Calculation. For the purposes of this study, 

C = Zc 
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where 2c is the total number of all the citations received over a two year period 
(including the year of publication) by all the publications (P) published by any 
researcher(s) officially affiliated with one of the Nutrition programs in the study. For 
example, the C for the 40 publications (P) published by the UFL’s Food Science and 
Nutrition Program in 1994 is 28 (see Table 7, p. 71). 

Data Source(s). For the purposes of this study, the publication data used to calculate C is 
drawn from the SCI-E section of ISI’s Web of Science citation database. The data is 
drawn for each year of the study 1992-1996, and for each publication from each Nutrition 
program used in this study. 

Interpretation. For the purposes of this study, C is considered to be an indicator research 
quality or significance of a Nutrition program’s P in terms of its impact or influence on 
subsequent scientific, scholarly research and publication (van Raan and van Leeuwen 
1998; van Raan 2000). 

Indicator FCSm 

Definition. For the purposes of this study, FCSm or Field Citation Score mean, is the 
world-wide or international mean citation rate for a two year period (including the year of 
publication) of all the publications P published by all of the scholarly journals in the ISI’s 
yC/?’s journal category for the field(s) in which a given Nutrition programs is actively 
researching and publishing (van Raan 2000). 

Calculation. For the purposes of this study. 
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FCSm= Ejif/2j 

where Ijif is the sum of all the JIFs for all the journals from each ISI Subject Category 
that corresponds to a field in which the Nutrition program actively researched and 
published in a given year, and £j is the total number of all the journals from each ISI 
Subject Category that corresponds to the field in which the Nutrition program actively 
researched and published in a given year. For example, the FCSm for the UFL’s Food 
Science and Nutrition Program for 1994 is calculated using the data from the 1996 JCR, 
with Ejif = 2041.255 and Ej = 914. The FCSm then is 2041.255/914 or 2.23 (see Table 7, 
p. 71). 

Note that FCSm is calculated for each year of the study using a two-year citation 
window for each journal in the field(s) in which the Nutrition program of interest is 
researching and publishing. Because the ISI calculates its JIF for each journal using a 
two-year citation window as well, FCSm can be derived by using the relevant JIFs from 
the JCR two years later than the particular year under study. For example, in the 
calculation above of the FCSm for the UFL’s Food Science and Nutrition Program for 
1994, the JIF data was drawn from the 1996 JCR. 

Data Source(s). For the purposes of this study, the journal publication data used to 
calculate the FCSm for each of the four Nutrition departments for each year was drawn 
from ISI’s JCR for two years later. That is, the data used to calculate the FCSm for the 
year 1992 was taken from the 199470?; to calculate the 1993 FCSm, the data was taken 
from the 1995 JCR, to calculate the 1994 FCSm, the data was taken from the 1996 JCR, 
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to calculate the 1995 FCSm, the data was taken from the 1997 JCR, and to calculate the 
1996 FCSm, the data was taken from the 1998 JCR. 

Interpretation. For the purposes of this study, FCSm provides a world-wide or 
international reference value for the comparison of average expected impact for the 
field(s) in which a given Nutrition program used in this study is active (van Raan and van 
Leeuwen 1998; van Raan 2000). 

Indicator JCSm 

Definition. For the purposes of this study, JCSm or Journal Citation Score mean, is the 
world-wide or international mean citation rate for a two year period (including the year of 
publication) of all the publications P published by all of the scholarly journals in a 
dynamic journal set composed of all the journals that contain at least one paper authored 
or co-authored by a member of a given Nutrition program in the same year. 

Calculation. For the purposes of this study, 

JCSm= ZjiFjs/2^jjs 

where Xjifjs is the sum of all the JIFs for all the journals in the journal set in which at 
least one member of a given Nutrition program published at least one publication in a 
given year, and Xjjs is the total number of all the journals in the journal set in which at 
least one member of a given Nutrition program published at least one publication in a 
given year. For example, the JCSm for the UFL’s Food Science and Nutrition Program 
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for 1994 is calculated using the JIF data from the 1996 JCR, with Ejifjs = 61.053 and Ijjs 
= 23. The JCSm then is 61.053/23 or 2.65 (see Table 7, p. 71). 

Note that the JCSm indicator is calculated using a two-year citation window for 
each journal in the field(s) in which the Nutrition program of interest is researching and 
publishing. Because the ISI calculates its JIF for each journal using a two-year citation 
window as well, JCSm can be derived by using the relevant JIFs from the JCR two years 
later than the particular year under study. For example, in the calculation above of the 
JCSm for the UFL’s Food Science and Nutrition Program for 1994, the JIF data was 
drawn from the 1996 JCR. 

Data Source(s). For the purposes of this study, the journal publication data used to 
calculate the JCSm for each of the four Nutrition departments for each year was drawn 
from ISFs JCR for two years later. That is, the data used to calculate the JCSm for the 
year 1992 was taken from the 1994 JCR, to calculate the 1993 JCSm, the data was taken 
from the 1995 JCR, to calculate the 1994 JCSm, the data was taken from the 1996 JCR, 
to calculate the 1995 JCSm, the data was taken from the 1997 JCR, and to calculate the 
1996 JCSm, the data was taken from the 1998 JCR. 

Interpretation. For the purposes of this study, JCSm provides a world-wide or 
international reference value for the comparison of average expected impact for a journal 
set in which a given Nutrition program used in this study publishes (van Raan and van 
Leeuwen 1998; van Raan 2000). 





54 



Indicator CPP 

Definition. For the purposes of this study, CPP is the average number of citations 
(including self-citations) per publication (P) (van Raan 2000) received for all of a given 
Nutrition program’s scholarly, scientific publications in any of the scholarly, scientific 
print journals covered by the ISFs Web of Science citation database and listed in its JCRs. 
Calculation. For the purposes of this study, 

CPP = Zc/Ep 

where Zp is the total number of all publications (P) published by any researcher(s) 
officially affiliated with one of the Nutrition programs in the study, and 2c is the total 
number of citations received by those same publications (P) over a two year period 
including the year of publication. For example, the CPP for the UFL’s Food Science and 
Nutrition Program for 1994 is calculated with Zc= 28 and Zp= 40. The CPP is then 28/40 
or 0.70 (see Table 7, p. 71). 

Data Source(s). For the purposes of this study, the publication and citation data used to 
calculate CPP is drawn from the SCI-E section of ISI’s Web of Science citation database. 
The data is drawn for each year of the study 1992-1996, and for each publication from 
each Nutrition program used in this study. 

Interpretation. For the purposes of this study, CPP is considered to be an indicator of the 
average actual impact of each publication by a given a Nutrition program in terms of that 
publication’s impact or influence on subsequent scientific, scholarly research and 
publication (van Raan and van Leeuwen 1998). 
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Indicator CPP/FCSm 

Definition. For the purposes of this study, CPP/FCSm is the average number of citations 
(including self-citations) per publication (CCP) received for all of a given Nutrition 
program’s scholarly, scientific publications in any of the scholarly, scientific print 
journals covered by the ISI’s Web of Science citation database and JCRs, compared with 
the FCSm or Field Citation Score mean, the world-wide or international mean citation 
rate for a two year period of all the publications P published by all of the scholarly 
journals in the ISI’s yC/?’s journal category for the field(s) in which the particular 
Nutrition program is actively researching and publishing. 

Calculation. For the purposes of this study, 

CPP/FCSm = S r/Zp 

Zjip/Sj 

where Zp is the total number of all publications (P) published by any researcher(s) 
officially affiliated with one of the Nutrition programs in the study; 2c is the total number 
of citations received by those same publications (P) over a two year period including the 
year of publication; Zjip is the sum of all the JIFs for all the journals from each ISI 
Subject Category that corresponds to a field in which the Nutrition program actively 
researched and published in a given year; and 2/ is the total number of all the journals 
from each ISI Subject Category that corresponds to a field in which the Nutrition 
program actively researched and published in a given year. For example, the CPP/FCSm 
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for the UFL’s Food Science and Nutrition Program for 1994 is calculated with Zc= 28, 
Ip= 40, IjiF= 2041.255, and Ij= 914. The CPP is then (28/40)7(204 1.25 5/9 14) or 
0.70/2.23 or 0.31 (see Table 7, p. 71) 

Note that the FCSm portion of the CPP/FCSm indicator is calculated using a two- 
year citation window for each journal in the field(s) in which the Nutrition program of 
interest is researching and publishing. Because the ISI calculates its JIF for each journal 
using a two-year citation window as well, the FCSm can be derived by using the relevant 
JTFs from the JCR two years later than the particular year under study. For example, in 
the calculation above of the FCSm for the CPP/FCSm for the UFL’s Food Science and 
Nutrition Program for 1994, the JIF data was drawn from the 1996 JCR. 

Data Source(s). For the purposes of this study, the data for calculating CPP/FCSm is 
drawn from the same sources as for the CPP and the FCSm indicators (see page and page 
respectively.) 

Interpretation. For the purposes of this study, CPP/FCSm is considered to be an 
indicator of the impact of a given Nutrition program’s research on the field(s) in which it 
is actively researching and publishing. If the CPP/FCSm result is > 1.0, then the 
Nutrition program’s work has a higher citation rate than the field-based world average 
(van Raan and van Leeuwen 1998; van Raan 2000). 
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Indicator CPP/JCSm 

Definition For the purposes of this study, CPP/JCSm is the average number of citations 
(including self-citations) per publication (CCP) received for all of a given Nutrition 
program’s scholarly, scientific publications in any of the scholarly, scientific print 
journals covered by the ISI’s Web of Science citation database and JCRs, compared with 
the JCSm or Journal Citation Score mean, is the world-wide or international mean 
citation rate for a two year period (including the year of publication) of all the 
publications P published by all of the scholarly journals in a dynamic journal set 
composed of all the journals that contain at least one paper authored or co-authored by a 
member of a given Nutrition program in the same year. 

Calculation. For the purposes of this study, 

CPP/JCSm = Sc/2p 

^JIFjs/^Jjs 

where Zp is the total number of all publications (P) published by any researcher(s) 
officially affiliated with one of the Nutrition programs in the study; Zc is the total number 
of citations received by those same publications (P) over a two year period including the 
year of publication; Zjifjs is the sum of all the JIFs for all the journals in the journal set in 
which at least one member of a given Nutrition program published at least one 
publication in a given year., and Zjjs is the total number of all the journals in the journal 
set in which at least one member of a given Nutrition program published at least one 
publication in a given year. For example, the CPP/JCSm for the UFL’s Food Science and 
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Nutrition Program for 1994 is calculated with Zc= 28, Zp= 40, ZjiFjs= 61.053 and Zjjs= 23. 
The CPP/JCSm then is (28/40)/(6 1.053/23) or 0.70/2.65 or 0.26 (see Table 7, p. 71). 

Note that the JCSm portion of the CPP/JCSm indicator is calculated using a two- 
year citation window for each journal in the field(s) in which the Nutrition program of 
interest is researching and publishing. Because the ISI calculates its JIF for each journal 
using a two-year citation window as well, the JCSm can be derived by using the relevant 
JIFs from the JCR two years later than the particular year under study. For example, in 
the calculation above of the JCSm for the UFL’s Food Science and Nutrition Program for 
1994, the JIF data was drawn from the 1996 JCR. 

Data Source(s). For the purposes of this study, the data for calculating CPP/JCSm is 
drawn from the same sources as for the CPP and the JCSm indicators (see page and page 
respectively.) 

Interpretation. For the purposes of this study, CPP/JCSm is considered to be an indicator 
of the impact of a given Nutrition program’s research on the journals in all the field(s) in 
which it is actively researching and publishing. If the CPP/JCSm result is > 1.0, then the 
Nutrition program’s work has a higher citation rate than the journal-based world average 
(van Raan and van Leeuwen 1998; van Raan 2000). 

Indicator JCSm/FCSm 

Definition. For the purposes of this study, JCSm/FCSm is the JCSm orJoumal Citation 
Score mean, the world-wide or international mean citation rate for a two year period 
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(including the year of publication) of all the publications P published by all of the 
scholarly journals in a dynamic journal set composed of all the journals that contain at 
least one paper authored or co-authored by a member of a given Nutrition program in the 
same year, compared with the FCSm or Field Citation Score mean, the world-wide or 
international mean citation rate for a two year period of all the publications P published 
by all of the scholarly journals in the ISI’syC/^’s journal category for the field(s) in 
which the particular Nutrition program is actively researching and publishing (Korevaar 
and Moed 1996). 

Calculation. For the purposes of this study, 

JCSm/FCSm = ZrTFjo/Sij g 

where Sjipjs is the sum of all the JIFs for all the journals in the journal set in which at 
least one member of a given Nutrition program published at least one publication in a 
given year., and Zjjs is the total number of all the journals in the journal set in which at 
least one member of a given Nutrition program published at least one publication in a 
given year; Zjif is the sum of all the JIFs for all the journals from each ISI Subject 
Category that corresponds to a field in which the Nutrition program actively researched 
and published in a given year; and Zj is the total number of all the journals from each ISI 
Subject Category that corresponds to a field in which the Nutrition program actively 
researched and published in a given year.. For example, the 1994 FCSm/JCSm for the 
UFL’s Food Science and Nutrition Program is calculated with Sjif = 2041 .255, Sj = 914, 
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SjiFjs = 61.053, and Zjjs = 23. The FCSm/JCSm then is (2041.255/914)/(61.053/23) or 
2.23/2.65 or 0.84 (see Table 7, p. 71). 

Note that both the JCSm and the FCSm parts of the JCSm/FCSm indicator is 
calculated using a two-year citation window for each journal in the field(s) in which the 
Nutrition program of interest is researching and publishing. Because the ISI calculates its 
JIF for each journal using a two-year citation window as well, both the JCSm and the 
FCSm can be derived by using the relevant JIFs from the JCR two years later than the 
particular year under study. For example, in the calculation above of the FCSm for the 
CPP/FCSm for the UFL’s Food Science and Nutrition Program for 1994, the JIF data was 
drawn from the 1996 JCR. 

Data Source(s). For the purposes of this study, the data for calculating FCSm/JCSm is 
drawn from the same sources as for the FCSm and the JCSm indicators (see page and 
page respectively.) 

Interpretation. For the purposes of this study, JCSm/FCSm is considered to be an 
indicator of the impact of a given Nutrition program’s journal set on the field(s) in which 
it is actively researching and publishing (Korevaar and Moed 1996;). For example, if the 
JCSm/FCSm result is > 1.0, then the Nutrition program’s journal set has a higher mean 
citation rate than that of all the papers “published in the fields to which the journals 
belong” (van Raan 2000, 3 1 7). 
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Journal set 

The decision to use a dynamic journal set rose from the problems experienced 
using a designated journal set in the preliminary study of the UTK Nutrition Department 
(Ackermann 2000). Early in the data collection phase of that study, it quickly became 
apparent that the designated journal set was not functioning as intended. It simply was 
not capturing all of the relevant data. If only the designated data set had been used, forty- 
one of the ninety-six publications (43%) and 343 of the 470 citations received (73%) by 
the UTK Nutrition Department during the five-year period under study would have been 
missed (Ackermann 2000). 

To remedy this problem, it was decided to use a dynamic journal set for this 
study. A dynamic journal set is composed of all the journals that contain at least one 
paper authored or co-authored by a member of a particular department in a given year. It 
has a composition that varies from year to year, reflecting the contemporary state of 
research productivity of a given department, capturing any changes in its publishing 
habits (Bourke and Butler 1993). Each dynamic journal set (referred hereafter as the 
journal set) then will form the journal sources from which publication (P) and citation 
(C) data will be gathered, as well as providing the journals from which the impact data 
will be drawn from the Journal Citation Reports (see Appendix E for the journal set data 
used in this study.) 
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Peer Institutions 

Three of the eight bibliometric indicators are ratios that contain built in 
comparisons of a department’s actual impact (CPP) with either an expected field-specific 
impact (FCSm) or a journal set-specific expected impact (JCSm), or compares a 
department’s journal set impact with its expected field impact (JCSm/FCSm). In all three 
of these cases, the comparisons are international averages for a given field or journal set. 
While this can be a useful measure, it is a high-level abstraction that is derived from the 
scholarly journal output of a large number of different kinds of research units that vary 
widely in attributes such as size of research staff, funding, as well as.the size and type of 
supporting or parent institution. 

Hence, it may be equally useful to compare research performance with a more 
compatible group of peer or benchmark institutions that share many of the same or 
similar features, such as size, level of funding, geographic location, etc. The UTK Office 
of Institutional Research and Assessment has compiled a list often peer institutions for 
use by the University of Tennessee-Knoxville in comparative performance evaluations 
(Office of Institutional Research and Assessment 2000) (see Table 4). Using these peer 
institutions will allow comparison to be made between the UTK Nutrition Department 
and similar programs in similar academic environments, thereby enhancing the accuracy 
and usefulness of the bibliometric assessment results (Martin and Irvine 1983; Martin 
1996; Zachos 1991). 
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Table 4. Peer, benchmark institutions used by the University of Tennessee-Knoxville for 
comparative performance evaluation purposes. 

University of Florida 

University of Georgia 

University of Kentucky 

University of Maryland at College Park 

University of North Carolina at Chapel Hill 

University of Oklahoma 

University of South Carolina 

University of Texas at Austin 

University of Virginia 

Virginia Polytechnic Institute and State University 

Source(s): 

Office of Institutional Research and Assessment. UTK’s peer institutions. 
<http://web.utk.edu/-~oira/peers.html>. 24 July 2000. 
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Although the peer institutions are reasonably comparable at the university level, it 
was not immediately evident that they all would be suitable for use in this study. In order 
to have an accurate and meaningful comparison at the department level, each of the peer 
institutions selected for use in this study must have a nutrition department, program, or 
research unit possessing approximately the same characteristics as the UTK Nutrition 
Department. To this end, each of the peer institutions was evaluated and rated using 
information gathered from each of their respective websites (for more detail, see 
Appendix C). Of the ten UTK peer institutions evaluated, two had no nutrition programs 
at all, and of the remaining eight institutions, the top three were selected for use in this 
study (see Table 5). Each of these three peer programs has an Extension Service 
component. 



Data Collection Methods 

The source for publication and citation data for each nutrition department for the 
five year period 1992-1996 is ISI’s Science Citation Index- Expanded (SCI-E), part of the 
Web of Science (2000) online database. The database was searched using the address 
data field in order to avoid as much as possible the problems associated with author 
identification such as homographs, misspelled names, etc. (MacRoberts arid MacRoberts, 
1989; Garfield 1989; Seglen 1997). It was found that a search argument composed of the 
most unique elements in a department’s institutional address was the most effective 
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Table 5. Selected UTK peer institutions in alphabetical order, by institution and 
department (or program) name. 
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Peer Institution 


Department (or Program) 


University of Florida 


Department of Food Science and Human Nutrition 


University of Georgia 


Department of Foods and Nutrition 


Virginia Polytechnic Institute 
and State University 


Department of Human Nutrition, Foods, and 
Exercise 



Source(s): 

University of Florida: http://fshn.ifas.ufl.edu/academic.htm; 
http://fshn.ifas.ufl.edu/research.htm; http://fshn.ifas.ufl.edu/faculty.htm 

University of Georgia: http://www.fcs.uga.edu/fdn/graduate; 
http://www.fcs.uga.edu/fdn/people/index.php3?FDN+faculty 

Virginia Polytechnic Institute and State University: 
http://www.chre.vt.edu/Admin/HNFE; 

http://www.chre.vt.edu/Admin/HNFE/fac/Davis/current_research.htm; 
http ://www. chre. vt . edu/j hwms/hnfegrad/faculty. html 
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(Ackermann 2000). For example, the search argument for the UTK Nutrition Department 
is “nutrition same tenn* same knoxville.” 

The source of the impact data for the five-year period 1994-1998 is ISI’s Journal 
Citation Reports (JCR). These reports are published annually, and are based on citation 
data for papers published two years previously (i.e., a two-year citation window) to allow 
time for the papers to be read, absorbed, and (possibly) cited by the scientific community. 
The data fi'om these sources will be recorded on data worksheet forms designed for this 
study (see Appendix D). 

Data Analysis Methods 

“Knowledge is the process of piling up facts; wisdom lies in their simplification.” 

- Martin Luther King, Jr. (Institute for Scientific Information 2000; Khalid 2000) 

The primary methods of data analysis will be descriptive, using tables and charts 
to find any patterns over time in the behavior of the bibliometric indicators. 

Limitations 

There are several limitations to this methodology that need to be kept in mind. 

The citation data is reported by calendar year, whereas most academic and government 
institutions run on a fiscal year, with the two rarely being synonymous. Due to the two- 
year publication lag between theJC/? updates, which reflects the nature of how citations 
accumulate, the most recent bibliometric indicators will always be at least two years 
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behind the current year. For example, if the 1998 JCR is the most current edition 
available, then the indicator values can only be current through 1996. 

There are minor differences between the journal title abbreviations used in the 
JCR and in the SCI-E. For example, the Journal of American Dietetic Association is 
abbreviated in the SCI-E as J Amer Diet Assn (Web of Science 2000), while listed in the 
JCR as J Am Diet Assoc (Institute for Scientific Information 1995). Unless otherwise 
indicated, the form used by the SCI-E will be the one used in this study. 

Only data from the citing journals are included in this study, whereas data from 
cited journals are not. This is due to the lack of impact and subject category data for 
cited journals in the JCR. For example, in the preliminary study (Ackermann 2000), two 
such journals were found containing publication and citation data from the UTK 
Nutrition Department, but were excluded from the study: the American Journal of 
Physiology, Endocrinology, and Metabolism', and Xh.Q Journal of Parenteral and Enteral 
Nutrition. 

The publication and citation data is drawn only from the relevant printed 
scholarly scientific literature currently found in the ISI citation databases. Since the ISI 
databases do not yet draw their data from electronic scholarly scientific journals, none 
will be included here. 
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CHAPTER 4 

DATA ANALYSIS AND EVALUATION 
Introduction 

Data was gathered for all eight bibliometric indicators, and the findings organized 
into data summaries (see Table 6 and Table 7). For purposes of analyzing these findings, 
each indicator was placed into one of three groups according to what that indicator was 
measuring. The indicator groups were the Publication Output Group, the Benchmark 
Impact Group, and the Actual Impact Group. Within each indicator group, the findings 
were analyzed to discover any patterns for the five-year period from 1992-1996, and then 
were placed in rank order comparisons based on aggregate five-year findings. The 
Publication Output Group contains only one indicator P. Two indicators, FCSm and 
JCSm, form the Benchmark Impact Group. The remaining five indicators are in the 
Actual Impact Group: C, CPP, CPP/FCSm, CPP/JCSm, and JCSm/FCSm. Note that the 
order in which the indicators are given above, P, FCSm, JCSm, C, CPP, CPP/FCSm, 
CPP/JCSm, and JCSm/FCSm, will be the same order in which they will appear in the 
data summary tables. 



Publication Output Group 

The pattern analysis of the publication output of the four Nutrition departments or 
programs under study reveled a mixed pattern of publication rates, three (UGA, UTK, 





Table 6. Data summary of bibliometric indicators for publication output (P), benchmark impact (FCSm, 
JCSm), and actual impact (C, CPP, CPP/FCSm, CPP/JCSm, JCSm/FCSm) by Nutrition department or 
program, 1992-1996. 
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Department 


Year 


or Program 


1992 


1993 


1994 


1995 


1996 


INDICATOR P: PUBLICATION OUTPUT 


Univ.Florida 


43. OC 


) 38.00 


ij 40.00 


l| 42.00 


j 37.00 


Univ. Georgia 


21.00 


37.00 


J 29.00 


36.00 


33.00 


Univ.Tenn-K 


17.00 


18.00 


18.00 


25.00 


18.00 


Virginia Tech 


9.00 


1 7.00 


1 8.00 


10.00 


11.00 


INDICATOR FCSm: MEAN IMPACT ON RESEARCH FEELDtS) 


Univ.Florida 


1.67 


2.08 


2.23 


1.78 


1.96 


Univ. Georgia 


2.15 


2.19 


2.59 


2.31 


1.95 


Univ.Tenn-K 


1.78 


1.35 


2.16 


1.92 


2.01 


Virginia Tech 


1.88 


0.91 


0.86 


1.61 


2.06 


INDICATOR J 


ICSm: MEAN IMPACT ON JOURNAL SET 


Univ.Florida 


5.34 


4.08 


2.65 


2.71 


3.09 


Univ. Georgia 


2.84 


3.71 


3.55 


3.21 


2.84 


Univ.Tenn-K 


2.94 


2.94 


3.92 


4.25 


3.24 


Virginia Tech 


3.31 


1.14 


1.15 


1.40 


2.91 


INDICATOR C: TOTAL ACTUAL IMPACT 


Univ.Florida 


30.00 


36.00 


28.00 


43.00 


41.00 


Univ. Georgia 


29.00 


16.00 


26.00 


21.00 


19.00 


Univ.Tenn-K 


23.00 


12.00 


5.00 


26.00 


13.00 


Virginia Tech 


9.00 


3.00 


2.00 


9.00 


4.00 


INDICATOR CPP: AVERAGE IMPACT PER PUBI 


-ICATION 


Univ.Florida 


0.70 


0.95 


0.70 


1.02 


1.11 


Univ.Georgia 


1.38 


0.43 


0.90 


0.58 


0.58 


Univ.Tenn-K 


1.35 


0.67 


0.28 


1.04 


0.72 


Virginia Tech 


1.00 


0.43 


0.25 


0.90 


0.36 


INDICATOR CPP/FCSm: AVERAGE IMPACT ON RESEARCH FIELDISt 


Univ.Florida 


0.42 


0.46 


0.31 


0.57 


0.57 


Univ.Georgia 


0.64 


0.20 


0.35 


0.25 


0.30 


Univ.Tenn-K 


0.76 


0.50 


0.13 


0.54 


0.36 


Virginia Tech 


0.53 


0.47 


0.29 


0.56 


0.17 
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Table 6. (continued) 



Department 
or Program 


Year 


1992 


1993 


1994 


1995 


1996 


INDICATOR CPP/JCSm: AVERAGE IMPACT ON JOURNAL SET 


Univ.Florida 


0.13 


0.23 


0.26 


0.38 


0.36 


Univ. Georgia 


0.49 


0.12 


0.25 


0.18 


0.20 


Univ.Tenn-K 


0.46 


0.23 


0.07 


0.24 


0.22 


Virginia Tech 


0.30 


0.38 


0.22 


0.64 


0.12 


INDICATOR JCSm/FCSm; AVERAGE JOURNAL SET IMPACT ON FIELDfSI 


Univ.Florida 


3.19 


1.96 


1.19 


1.52 


1.58 


Univ. Georgia 


1.32 


1.69 


1.37 


1.39 


1.46 


Univ.Tenn-K 


1.65 


2.18 


1.81 


2.21 


1.61 


Virginia Tech 


1.76 


1.25 


1.34 


0.87 


1.41 



Note(s): 

Field or research field is all the ISI Subject Categories that contain at least one journal that published at 
least one paper by at least one researcher fiom one of the departments or programs under study. 

Sourcefs): 

Institute for Scientific Information. 1995. 1994JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia: Institute for .Scien tifi c 
Information Press. 

Institute for Scientific Information. 1996. 1995 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia; Instit u t e for Scientific 
Information Press. 

Institute for Scientific Information. 1997. 1996 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia: I nstitute for Scientific 
Information Press. 

Institute for Scientific Informatioa 1998. 1997 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia: Institute for Scientific 
Information Press. 

Institute for Scientific Informatioa 1999. 1998 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia; Institute for Scientific 
Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia; Institute for Scientific Information. 
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Table 7. Data summaiy for each Nutrition department or program by bibliometric indicators for publication 
output (P), benchinark impact (FCSm, JCSm), and actual impact (C, CPP, CPP/FCSm, CPP/JCSm, 

JCSm/FCSm), 1992-1996. 





Year 


Indicator 


1992 


1993 


1994 


1995 


1996 


DEPT. OF FOOD SCIENC 


& HUMAN NUTRITION, UNIV. OF FLORIDA 


P 


43.00 


38.00 


40.00 


42.00 


37.00 


FCSm 


1.67 


2.08 


2.23 


1.78 


1.96 


JCSm 


5.34 


4.08 


2.65 


2.71 


3.09 


C 


30.00 


36.00 


28.00 


43.00 


41.00 


CPP 


0.70 


0.95 


0.70 


1.02 


1.11 


CPP/FCSm 


0.42 


0.46 


0.31 


0.57 


0.57 


CPP/JCSm 


0.13 


0.23 


0.26 


0.38 


0.36 


JCSm/FCSm 


3.19 


1.96 


1.19 


1.52 


1.58 


DEPT. OF FOODS & NUTRITION, UNIVERSITY OF GEORGIA 


P 


21.00 


37.00 


29.00 


36.00 


33.00 


FCSm 


2.15 


2.19 


2.59 


2.31 


1.95 


JCSm 


2.84 


3.71 


3.55 


3.21 


2.84 


C 


29.00 


16.00 


26.00 


21.00 


19.00 


CPP 


1.38 


0.43 


0.90 


0.58 


0.58 


CPP/FCSm 


0.64 


0.20 


0.35 


0.25 


0.30 


CPP/JCSm 


0.49 


0.12 


0.25 


0.18 


0.20 


JCSm/FCSm 


1.32 


1.69 


1.37 


1.39 


1.46 


DEPT. OFNLT 


nUTION, UNIVERSITY OF TENNESSEE-KNOXVILLE 


P 


17.00 


18.00 


18.00 


25.00 


18.00 


FCSm 


1.78 


1.35 


2.16 


1.92 


2.01 


JCSm 


2.94 


2.94 


3.92 


4.25 


3.24 


C 


23.00 


12.00 


5.00 


26.00 


13.00 


CPP 


1.35 


0.67 


0.28 


1.04 


0.72 


CPP/FCSm 


0.76 


0.50 


0.13 


0.54 


0.36 


CPP/JCSm 


0.46 


0.23 


0.07 


0.24 


0.22 


JCSm/FCSm 


1.65 


2.18 


1.81 


2.21 


1.61 


DEPT. OF HUMAN NUTRITION, FOODS, & EXERCISE, VIRGINIA TECH 


P 


9.00 


7.00 


8.00 


10.00 


11.00 


FCSm 


1.88 


0.91 


0.86 


1.61 


2.06 


JCSm 


3.31 


1.14 


1.15 


1.40 


2.91 


C 


9.00 


3.00 


2.00 


9.00 


4.00 


CPP 


1.00 


0.43 


0.25 


0.90 


0.36 


CPP/FCSm 


0.53 


0.47 


0.29 


0.56 


0.17 


CPP/JCSm 


0.30 


0.38 


0.22 


0.64 


0.12 


JSm/FCSm 


1.76 


1.25 


1.34 


0.87 


1.41 
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Table 7. (continued) 

Note(s): 

Field or research field is all the ISI Subject Categories that contain at least one journal that published at 
least one paper by at least one researcher from one of the departments or programs under study. 

Source(s): 

Institute for Scientific Informatioa 1995. 1994 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia; Institute for Scientific 
Information Press. 



Institate for Scientific Informatioa 1996. 1995 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. y\a\&ds\p\aa\ Institute for Scientific 
Information Press. 

Institute for Scientific Information. 1997. 1 996 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia: Institute for Scientific 
Information Press. 



Institute for Scientific Information. 1998. 1997 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia: Institute for Scientific 
Information Press. 



Institute for Scientific Informatioa 1999. 1998 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia: Institute for Scientific 
Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia; Institute for Scientific Information. 
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Figure 1 . P or publication output or the total number of journal publications for each 
Nutrition program, 1992-1996. 
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and VT) ending with a higher yearly total in 1996 than in 1992 (see Table 6, Table 7, and 
Figure 1). The exception was UFL, which ended with a lower publication rate in 1996 
than in 1992. All four departments experienced their greatest increase in publications 
between 1993 and 1995, followed by a decrease in 1996 for the programs at UFL, UGA 
and UTK. VT’s program had the steadiest pattern, characterized by a slight decrease in 
total publications from 1992 to 1993, followed by a slow, steady increase from 1993 to 
1996. The most volatile pattern was shown by UGA, which was characterized by sharp 
upswings in 1993 and 1995, and sharp downswings in 1994 and 1996. An examination 
of the rank order of the four Nutrition programs based on the five-year aggregate findings 
shows that UFL ranked first with 40% of the total publications (N= 200), followed by 
UGA with 3 1% (N= 156), UTK with 20% (N= 96), and VT with 9% (N= 45) (see Table 
8 ). 



Benchmark Impact Group 

The five-year pattern for the Benchmark Impact Group will examined separately 
for each of the two indicators in the group, FCSm and JCSm. Patterns will be analyzed 
first for both indicators, followed by the rank order comparisons for each Nutrition 
program by indicator. From 1992 to 1993, the FCSm indicator for both UTK and VT 
declined while the FCSm for UFL and UGA increased (see Table 6, Table 7, and Figure 
2). From 1993 to 1994 saw the FCSm for UFL, UGA, and UTK increase to its highest 
point, while the 
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Table 8. Rank order of Nutrition departments or programs publication output (P), mean impact (FCSm, 
JCSm), and actual impact (C, CPP, CPP/FCSm, CPP/JCSm, JCSm/FCSm) for the aggregate five-year 
period 1992-1996. 



Department in 


Five-year Aggregate 


Department in 


Five-year Aggregate 


Rank Order 


Total (N) 


Mean 


Percent 


Rank Order 


Total (N) 


Mean 


Percent 


INDICATOR P: 






INDICATOR CPP. 






PUBLICATION OUTPUT 






AVERAGE IM 


PACT PER PUBLICATION 


Univ. Florida 


200.0C 


40.00| 40®/c 


Univ. Florida 


4.4{ 


i 0.9C 


1 29®/, 


Univ. ^Georgia 


156.0C 


31.20 31®/, 


Univ. Tenn-K 


4.0t 


3 0.81 


27®/, 


Univ. Tenn-K 


96.0C 


19.20 20®/, 


Univ. Georgia 


3.8-: 


n 0.77 


25®/, 


Virginia Tech 


45.00 


9.00| 09®/, 


Virginia Tech 


2.94 


J 0.59 


19®/, 


Total 


497.00 


99.40| 100®/, 


Total 


15.3f 


a 3.07 


100®/, 


INDICATOR FCSm: MEAN IMPACT 


INDICATOR CPP/FCSm: AVERAGE 


OF THE RESEARCH FIELD(S) 




IMPACT ON RESEARCH FIELD(S) 


Univ. Georgia 


11.19 


2.24 


30®/, 


Univ. Florida 


2.33 


0.47 


28®/, 


Univ. Florida 


9.72 


1.94 


26®/o 


Univ. Tenn-K 


2.29 


0.46 


27®/, 


Univ. Tenn-K 


9.22 


1.84 


24®/« 


Virginia Tech 


2.02 


0.40 


24®/, 


Virginia Tech 


7.32 


1.46 


20% 


Univ. Georgia 


1.74 


0.35 


21®/, 


Total 


37.45 


7.48 


100% 


Total 


8.38 


1.68 


100®/, 


INDICATOR JCSm: 






INDICATOR CPP/JCSm: AVERAGE 


MEAN IMPACT OF THE JOURNAL SET 


IMPACT ON THE JOURNAL SET 




Univ. Florida 


17.87 


3.57 


30®/, 


Virginia Tech 


1.66 


0.33 


30®/, 


Univ. Tenn-K 


17.29 


3.46 


28®/, 


Univ. Florida 


1.36 


0.27 


25®/, 


Univ. Georgia 


16.15 


3.23 


26®/, 


Univ. Georgia 


1.24 


0.25 


23®/, 


Virginia Tech 


9.91 


1.98 


16®/, 


Univ. Tenn-K 


1.22 


0.24 


22®/, 


Total 


61.22 


12.24 


100®/, 


Total 


5.48 


1.09 


100®/, 


INDICATOR C: 






INDICATOR JCSm/FCSm: AVERAGE 


TOTAL ACTUAL IMPACT 






JOURNAL SET IMPACT ON FIELD(S) 


Univ. Florida 


178.00 


35.60 


45®/, 


Univ. Tenn-K 


9.46 


1.89 


29®/, 


Univ. Georgia 


111.00 


22.20 


28®/, 


Univ. Florida 


9.44 


1.89 


29®/, 


Univ. Tenn-K 


79.00 


15.80 


20®/, 


Univ. Georgia 


7.23 


1.45 


22®/, 


Virginia Tech 


27.00 


5.40 


07®/, 


Virginia Tech 


6.63 


1.33 


20®/, 


Total 


395.00 


79.00 


100®/, 


Fotal 


32.76 


6.56 


100®/, 



Note(s): 

Field(s) or research field(s) is all the ISI Subject Categories that contain at least one journal that published 
at least one paper at least one researcher from one of the departments or programs under study. 
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Figure 2. FCSm or mean impact of journal publications of the research field(s) for each 
Nutrition program, 1992-1996. Field(s) = all ISI Subject Categories that contain at least 
one journal that published at least one paper by at least one researcher. 
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FCSm for VT decreased to its lowest point. During the two years from 1994 to 1996, 
UGA’s FCSm declined to its lowest point, ending below its 1992 value, while both 
UFL’s and UTK’s FCSm declined then recovered, ending slightly above their respective 
1992 values. VT’s FCSm showed a sharp increase to end slightly higher than its 1992 
value. The rank order of the Nutrition departments by FCSm ran from UGA with 30% 
(N= 1 1. 19) of the aggregate value, followed by UFL with 26% (N= 9.72), UTK with 24% 
(N= 9.22), and VT with 20% (N= 7.32) (see Table 8). 

The JCSm indicator for UFL experienced a steep, continuous decrease from 1992 
to 1994, followed by a slight increase through 1996, ending lower than its 1992 value 
(see Table 6, Table 7, and Figure 3). The JCSm for UGA increased from 1992 to 1993, 
followed by a steady decline to 1996, ending with the same value as in 1992. The JCSm 
for UTK increased overall from 1993 to 1996, ending above its 1992 value. VT’s JCSm 
dropped precipitously from 1992 to 1993, then increased steadily to 1996, ending below 
its 1992 values. UFL lead in the rank order by JCSm for the four Nutrition programs 
with 30% (N= 17.87) of the aggregate value, followed by UTK with 28% (N= 17.29), 
UGA with 26% (N= 16.15), and VT with 16% (N= 9.91) (see Table 8). 

Actual Impact Group 

The Actual Impact Group consists of five indicators: C, CPP, CPP/FCSm, 
CPP/JCSm, and JCSm/FCSm. As was done with the indicators in the Expected Impact 
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Figure 3. JCStn or mean impact of the journal set in which a Nutrition program 

publishes, 1992-1996. 
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Group, the five-year pattern will be examined separately for each indicator in the group, 
followed by the rank order comparisons for each Nutrition program by indicator. 

The patterns for C for all four Nutrition departments was characterized by a 
general decline in its overall values from 1992 to 1996, ending with values below those 
of 1992, despite having an upswing year in either 1994 or 1995 (see Table 6, Table 7, and 
Figure 4). The exception was UFL, which experienced an overall net increase in C from 

1992-1996. The rank order by C value for the departments has UFL first with 45% (N= 

1 

178) of the total citations received, followed by UGA in second place with 28% (N= 

111), UTK in third place with 20% (N= 79), and VT in last place with 7% (N= 27) (see 
Table 8). 

The indicator CPP had a pattern similar to that for C (see Table 6, Table 7, and 
Figure 5). All the Nutrition departments except UFL experienced a general, overall 
decline from 1992 to 1994, with UTK and VT experiencing a recovery from 1994 to 
1995 (1993 -1994 for UGA), and with all three (UGA, UTK, and VT) ending with values 
in 1996 below the 1992 mark. The exception was UFL, which experienced an overall 
increase, ending the five-year period with a CPP well above that for 1992. The rank 
order by CPP value shows UFL with the highest part of the aggregate value at 29% 
(N=4.48), followed by UTK with 27% (N=4.06), UGA with 25% (N= 3.87), and VT with 
19% (N= 2.94) (see Table 8). 

The CPP/FCSm indicator was below average (<1.0) for all the departments for the 
entire period (see Table 6, Table 7, and Figure 6). All four programs (UFL, UTK, VT, 
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Figure 4. C or total number of citations to journal publications for each Nutrition 

program, 1992-1996. 
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Citations per publication (CPP) 
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—•—Univ. Florida — Univ.Georgia — A— Univ.Tenn-K Virginia Tech 



Figure 5. CPP or average actual impact of journal publications for each Nutrtion 
program, 1992-1996. CPP = citations per publication. 
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Actual field impact (CPP/FCSm) 




Note(s): Field= all the ISI Subject Categories that contain at least one journal 
that published at least one paper by at least one researcher. 

— Univ.Florida — Univ.Georgia — A— Univ.Tenn-K —X— Virginia Tech 

Figure 6. CPP/FCSm or average impact of journal publications from each Nutrition 
program on the field(s) in which it researches, 1992-1996. For comparative purposes, 
the mean expected impact = 1.0. Hence, <1.0 is below average, >1.0 is above average. 
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and UGA) had a uniform decline in CPP/FCSm value from 1992 to 1994, followed by an 
upswing from 1994 to 1995. From 1995 to 1996, the CPP/FCSm value for UFL 
remained steady through 1996, while that for VT and UTK decreased and UGA 
increased, with all the programs still ending with 1996 values lower than those did for 
1992. The exception was UFL, which posted higher values for 1996 than for 1992. The 
rank order of Nutrition departments by CPP/FCSm value had UFL in first place with 28% 
(N= 2.33) of the aggregate value. The CPP/FCSm values for the remaining three 
departments were within 7% of each other: UTK with 27% (N= 2.29), VT with 24% (N= 
2.02), and UGA with 21% (N= 1.74) (see Table 8). 

The indicator CPP/JCSm had a general pattern very similar to that for 
CPP/FCSm. All values were below average (<1.0), with sharp increases from 1994 to 
1995, followed by declines, ending in 1996 with values less than those for 1992 (see 
Table 6, Table 7, and Figure 7). UFL’s CPP/JCSm was the exception again, showing an 
overall increase from 1992 to 1996, ending above its 1992 value. VT led the CPP/JCSm 
rank order with 30% (N= 1 .66) of the aggregate value. The remaining three departments 
in rank order were, UFL with 25% (N= 1.36), followed by UGA with 23% (N= 1.2.4) 
and UTK with 22% (N= 1.22). 

The last actual impact indicator, JCSm/FCSm, was above average (>1.0) for each 
department for the 1992-1996 period, with the sole exception of VT in 1995 (see Table 6, 
Table 7, and Figure 8). UFL’s JCSm/FCSm experience the most precipitous decline 
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Actual journal set impact (CPP/JCSm) 



84 * 




— Univ.Florida Univ. Georgia — A— UnIv.Tenn-K - K Virginia Tech 



Figure 7. CPP/JCSm or average impact of journal publications fiom each Nutrition program on 
the journal set in which it publishes, 1992-1996. For comparative purposes, the mean expected 
impact “ 1 .0. Hence, <1 .0 is below average, >1 .0 is above average. 




97 



Journal set's field impact (JCSm/FCSm) 
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journal that published at least one paper by at least one researcher. 

— Univ.Florida — Univ.Georgia -tA— U niv.Tenn-K —»<— Virginia Tech 

Figure 8. JCSm/FCSm or average journal set impact for journals published in the 
field(s) for each Nutrition program, 1992-1996. For comparative purposes, the 
mean expected impact = 1.0. Hence , <1.0 is below average and >1.0 is above 

average. 
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from 1992 to 1994, after which it rallied slightly for the remaining two years, ending well 
below its 1992 value. UGA’s values rose slightly from 1992 to 1993, then dropped in 
1994, rising through 1996, but ending above its 1992 value. UTK experienced an 
increase in its JCSm/FCSm values from 1992 to 1993, a decline from 1993 to 1994, 
followed by an increase from 1994 to 1995, and ending in 1996 slightly below its 1992 
value. VT’s JCSm/FCSm was above average for the entire five-year period, with the 
exception of 1995. Even so VT’s values underwent a steady decline from 1992 to 1995, 
followed by a recovery from 1995 to 1996, ending below its 1992 value. The rankings by 
JCSm/FCSm showed UTK and UFL tied with the highest percentage of the aggregate 
value at 29% (N= 9.46 and N= 9.44 respectively). UGA was ranked second with 22% of 
the aggregate value (N= 7.23), followed by VT with 20% (N= 6.63) (see Table 8). 

Summary of Findings 

In general, the Publication Output Group ended with a higher number of total 
publications in 1996 than it began with in 1992. The FCSm indicator from the 
Benchmark Impact Group had 1996 values only slightly above or slightly below their 
1992 values. The JCSm indicator values tended to end with 1996 values at or below their 
1992 beginning values. The indicators in the Actual Impact Group shared a similar 
pattern of value changes for UGA, UTK, and VT, with decreases from 1 992 to 1 994, 
increases from 1994 to 1995, followed by decreases again from 1995 to 1996, all ending 
below the 1992 values. For four of the five indicators (C, CPP, CPP/FCSm, and 
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CPP/JCSm), UFL was the exception, ending with 1996 values greater than those for 
1992. 

The overall rank order results, when averaged by department, ranged 
approximately between one and four. UFL had the highest average ranking of 1 .4, 
followed by UTK with 2.5, UGA with 2.6, and VT with 3.5. This rank order remains the 

same when examining just the Actual Impact Group of indicators. The rank order 

/ 

changes slightly to UFL, UGA, UTK, and VT when examining the Publication Output 
Group and the Benchmark Group of indicators. These average rank order results should 
be treated with caution however as they were created from values for indicators that are 
only somewhat related, not directly comparable entities. 

Discussion of Findings 

The overall rank order of UFL, UTK, UGA, and VT may be in part reflective of 
the relative differences in the size of the researcher staff available in each program. The 
UFL program has the largest staff with 32 researchers, followed by UGA and VT with 24 
researchers each, and UTK with 17 (see Appendix C). However, the differences in 
staffing cannot explain everything as UTK, with the smallest research staff of the four 
Nutrition programs, handily out performed VT and surpassed UGA (albeit slightly), two 
programs possessing larger research staffs. Also, if staff size alone was a major 
determinant of performance rank order, then VT’s and UGA’s programs, each staffed 
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with the same number of researchers, should have consistently ranked more closely 
together than they actually did (see Table 8). 

It is interesting to note that for four of the five Actual Impact Group indicators, C, 
CPP, CPP/FCSm, and CPP/JCSm, UFL was the only program that did not experience an 
overall decline from 1992 to 1996 (see Table 6 and Figures 4-7). This means that the 
UFL Nutrition program produced publications that had a greater impact on the fields in 
which it conducted research and on the journals in which it published than did the other 
three Nutrition programs used in this study. As for the fifth indicator of actual impact 
JSCm/FCSm, the four Nutrition programs in this study showed mixed results. Two 
programs, UFL and VT saw an overall decline in the impact of the journals in which they 
published their research results, whereas UGA experienced an increase and UTK remain 
about the same. That is, on the average, researchers from two (UFL and VT) of the four 
programs were getting their research published in journals with lower impact factors in 
1992 than they were in 1996. One program, UGA, was getting its research published in 
higher impact journals, while the impact of the journals that published the UTK research 
remained relatively unchanged overall. 

It is also interesting to note that the rank order for the P and C indicators are the 
same (see Table 8). This suggests that the quantity of publications (P) produced by a 
given Nutrition department may have a degree of influence on the quantity of citations 
(C) that its publications receive. That is, the greater a Nutrition program’s P, the greater 
its C. At one level, this relationship may simply be the result of probability, with the 
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greater the number of publications given a Nutrition department produces, the greater the 
number of citable papers that exist in the literature to be cited, hence the greater the 
possibility of being cited. 

There is some indication in the literature that the benchmark indicators FCSm and 
JCSm can have a certain predictive quality. That is, by implication they provide a 
comparative expected value against which an actual result, such as CPP, is compared 
(van Raan and van Leeuwen 1998). Therefore, it can be assumed that FCSm functions as 
a predictor of CPP/FCSm or the actual impact on a given program’s research field(s), and 
JCSm as the predictor of CPP/JCSm or the actual impact on the journal set in which a 
given program publishes. For example, in this study, if one used the five-year average 
value of FCSm to predict the CPP/FCSm rankings, the results should find UGA ranked 
first, followed by UFL, UTK, and VT. Similarly, using the five-year average values of 
the JCSm indicator to predict the CPP/JCSm rankings, the results should be UFL ranked 
first, followed by UTK, UGA, and VT (see Table 8). The data from this study, however, 
does not lend much support to this assumption. Of all the rankings predicted by both the 
FCSm and JCSm indicators using five-year data averages, only one ranking occurred as 
predicted, UGA’s JCSm prediction of a third ranking for its CPP/JCSm (see Table 8 and 
9). When examined on a year-by-year basis, the predictive capabilities of FCSm and 
JCSm do not seem to improve. The FCSm rank order predicted the CPP/FCSm rank 
order correctly only three times, ranking UFL fourth in 1992, UGA and UFL first and 
second respectively in 1994, and UTK second in 1996. The JCSm indicator preformed 
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even more poorly by only correctly predicting two rankings for CPP/JCSm, which were a 
third rank for UTK in 1993 and a second rank for UGA in 1994 (see Table 9). 
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Table 9. Rank order prediction of field impact (FCSm vs. CPP/FCSm) and journal set 
impact (JCSm vs. CPP/JCSm), by year and by five-year average, 1992-1996. 
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Comparison by year of: 


FCSm (predicted rank order) vs. CPP/FCSm (actual rank order) 


1992 


1993 


1994 


1995 


1996 


FCSm 


CPP/ 

FCSm 


FCSm 


CPP/ 

FCSm 


FCSm 


CPP/ 

FCSm 


FCSm 


CPP/ 

FCSm 


FCSm 


CPP/ 

FCSm 


UGA 


UTK 


UGA 


UTK 


UGA 


UGA 


UGA 


UFL 


VT 


UFL 


VT 


UGA 


UFL 


VT 


UFL 


UFL 


UTK 


VT 


UTK 


UTK 


UTK 


VT 


UTK 


UFL 


UTK 


VT 


UFL 


UTK 


UFL 


UGA 


UFL 


UFL 


VT 


UGA 


VT 


UTK 


VT 


UGA 


UGA 


VT 


JCSm (predicted rank order) vs. CPP/JCSm (actual rank order) 


1992 


1993 


1994 


1995 


1996 


JCSm 


CPP/ 

JCSm 


JCSm 


CPP/ 

JCSm 


JCSm 


CPP/ 

JCSm 


JCSm 


CPP/ 

JCSm 


JCSm 


CPP/ 

JCSm 


UFL 


UGA 


UFL 


VT 


UTK 


UFL 


UTK 


VT 


UTK 


UFL 


VT 


UTK 


UGA 


UFL 


UGA 


UGA 


UGA 


UFL 


UFL 


UTK 


UTK 


VT 


UTK 


UTK 


UFL 


VT 


UFL 


UTK 


VT 


UGA 


UGA 


UFL 


VT 


UGA 


VT 


UTK 


VT 


UGA 


UGA 


VT 



Comparison by five-year average of 


FCSm (predicted rank order) vs. 
CPP/FCSm (actual rank order) 


JCSm (predicted rank order) vs. CPP/JCSm 
(actual rank order) 


FCSm 


CPP/FCSm 


JCSm 


CPP/JCSm 


UGA 


UFL 


UFL 


VT 


UFL 


UTK 


UTK 


UFL 


UTK 


VT 


UGA 


UGA 


VT 


UGA 


VT 


UTK 



Note(s): 

The bold entries represent correctly predicted rank orders. 




104 



92 



CHAPTER 5 

CONCLUSIONS AND RECOMMENDATIONS 

“I never thought that others would take them more seriously than I did ” 

- Albert Einstein, talking about his theories 
(Institute for Scientific Information 2000; Khalid 2000) 

Conclusions of the Study 

The goal of this study was to develop a set of empirically derived and 
theoretically sound citation-based bibliometric indicators of scientific research 
performance and apply them in an exploratory study comparing the University of 
Tennessee-Knoxville’s Nutrition Department with three of its peer programs at the 
University of Florida, the University of Georgia, and Virginia Tech over a five-year 
period (1992-1996). A set of eight bibliometric indicators were developed and used in 
this evaluation: P, FCSm, JCSm, C, CPP, CPP/FCSm, CPP/JCSm, and JCSm/FCSm (see 
Tables.) 

Based on the findings of this study, the UTK Nutrition Department was ranked 
third in publication or research output (P), third in the average impact of its published 
papers on its research field(s) (FCSm), and second in the average impact of its 
publications on the journal set in which it published (JCSm) (see Table 8). In terms of 
the actual impact of its publications, UTK’s Nutrition Department ranked third in total 
citations received by all its published journal articles (C), second in the average impact of 
its journal publications (CPP), second in the average impact of its journal publications on 
the field(s) in which the department conducted research (CCP/FCSm), fourth in the 
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average impact of its journal publications had on the set of journals in which the 
department published (CPP/JCSm), and first in the average impact that these journals had 
on the field(s) in which the department conducted its research (JCSm/FCSm) (see Table 
8). The overall average ranking for the UTK Nutrition Department in this study was 
second. 



Interpretative Limitations of Bibliometric Evaluations 
The important thing to remember when drawing conclusions from bibliometric 
evaluations of research performance is that their results are primarily descriptive and can 
offer little or nothing in the way of causal explanations. That is, they can describe 
patterns in publication and citation, the relative impact ranking between journals, and 
show any increases and decreases over time for various measures of departmental 
research performance, but cannot say much (if anything) about why the results are they 
way they are or how they came about. Such causal explanations require more intimate 
and detailed knowledge about each of the programs involved, such as the past and current 
research expertise and productivity of each researcher, projects completed, projects in 
progress, current funding sources, size and functionality of existing laboratory facilities, 
and number and quality of graduate students, than can be gleaned from the publication 
and citation databases. An understanding of the Nutrition discipline in general and the 
state of the art over the time period studied is also important for the formulation of useful 
and relevant causal explanations of the results of bibliometric evaluations. For example. 
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this study demonstrated a pattern of total citations received (C) by each of the four 
Nutrition departments over a five-year period (see Figure 4), but cannot explain why this 
pattern exists or how it came about. Was the overall decline in citations received by three 
of the programs under study (UGA, UTK, and VT) due to the loss of key researchers by 
the departments through transfer or retirement, or due to the ending of a large project 
with no replacement yet in place, or due to natural causes such a flood, fire or 
earthquake? Alternatively, was the overall decline due instead to changes within the 
research priorities of the Nutrition field as a whole, with a shifting of emphasis to 
subjects not covered by the expertise extant in the departments under study? This would 
lead to a decline in the citation rate of journal publications that were no longer in the 
“hot” or even mainstream research areas. Based only the information found in this study, 
how is one to know which (if any) of these explanations are correct or even applicable? 

These findings about the relative research performance of the four Nutrition 
programs under examination can be useful as a management tool for deans or department 
heads of these departments as well as to the Dean of UTK’s College of Human Ecology 
in particular. In addition, the results of this study can make a useful contribution to the 
fields of scientometrics and infometrics by the through the development of a set of 
empirically and theoretically robust of bibliometric indicators. 
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Bibliometric Results as a Management Tool 
The results of this study demonstrate the standing of the UTK Nutrition 
Department relative to three of its peer departments, as well as relative to all the Nutrition 
research units represented in the ISI citation database, in terms of the eight bibliometric 
measures used to evaluate research performance (see Table 8). The Dean of the UTK 
College of Human Ecology (CHE), as well as the deans and department heads of the 
other three programs, could use these results as additional inputs into the planning 
process in order to develop the best strategies for maximizing the department’s strengths 
and minimizing its weaknesses. In addition, the Dean of the CHE could make use of the 
findings in making management decisions, such as whether or not to hire more research 
faculty or recruit more graduate students, and as additional data useful for helping secure 
additional funding from granting agencies as well as from the university administration. 

The Dean may also find that the results of the four bibliometric indicators used in 
this study that had a strong relationship with peer review findings (CPP, CPP/FCSm, 
CPP/JCSm, and JCSm/FCSm) most useful for on-going ranking purposes when 
integrated with any current and future peer review data that he has for the UTK Nutrition 
Department. The findings of this study could also be utilized as the basis for developing 
an on-going, in-house database for tracking the current and future research performance 

of the UTK Nutrition Department against that of its specified peer programs as well 

1 

against that of the Nutrition field as a whole. 
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Contributions to Scientometrics and Infometrics 

This study provides a needed addition to the literature of departmental level 
bibliometric studies by providing both a source of comparative data and a set of 
bibliometric indicators of known empirical and theoretical utility for application in other 
studies in similar contexts. This study also may provide further empirical data on the 
behavior and utility of specific bibliometric indicators either previously used in the 
literature, or previously used in the literature but modified with ISI’s impact data for use 
in this study. 

Of particular interest in this respect are the four indicators CPP, CPP/FCSm, 
CPP/JCSm, and JCSm/FCSm that have the strongest empirical and theoretical rationale 
of those examined in this study. These indicators are all primary indicators that are 
shown to correlate with other measures of scientific achievement such as peer review 
ranking, awards, and academic positions (see Appendix B). In addition, CPP, 
CPP/FCSm, CPP/JCSm, and JCSm/FCSm are all contextualized (i.e., relative or 
comparative) measurement types that provide their own intrinsic, meaningful context that 
gives the strongest potential explanatory power (see pp. 43-45; van Raan 2000). In the 
future, these four indicators may form the basis of what will become a standard set of 
recognized, core indicators that must be present in every well-conducted departmental 
level bibliometric study. 

The successful (if limited) application of Martin & Irvine’s (1981, 1983) 
methodology of multiple converging partial indicators in this study will hopefully 
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encourage others conducting departmental level bibliometric research performance 
studies to use more than two or three bibliometric indicators, thereby improving the 
accuracy and quality of their results (Martin, 1996). 

Recommendations for Further Research 
Further Subject Development 

In light of the use of Martin and Irvine’s (1983) methodology of multiple, 
converging indicators, it is worth considering what could be added to this study to 
explain more about the measurement of academic research performance in the Nutrition 
field. Some of the areas meriting further exploration would include departmental 
publication habits, peer review ratings, nature of each department’s research activity, the 
inclusion of non-citation based indicators of scientific achievement, and the peer 
departments or programs that have been used previously for benchmarking purposes by 
each of the Nutrition programs under study. 

Departmental publications habits 

The publication habits of each Nutrition program under study could be further 
examined in order to accurately gauge how much of its research output is actually 
covered, and hence explainable by, the findings drawn from only the consideration of the 
traditional, scholarly print journal literature. For example, for each department or 
program, how much of the research is published in alternate sources such as books and 
monographs, in the “gray literature” (e.g., technical reports and patents), or in journals 
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not covered by the SCI-E, as well as in other related areas, such as education, that may be 
covered in the SSCI or AHCI hut not in the SCI-E. It would also be interesting to 
discover to what degree scholarly, scientific departmental publishing is moving from 
print to electronic outlets, such as peer reviewed (or not) electronic journals, as well as 
appearing in web sites created and maintained primarily by either the author, or a 
professional organization or society. 

Peer review data 

The addition of any existing peer review data available for each Nutrition 
department or program under study would add another indicator of research performance. 
In this case, the data would provide a qualitative dimension to an otherwise quantitative 
assessment, while remaining relevant due to the previously reported correlations with the 
indicators used in the study (Lawani and Beyer 1983; McAllister, Anderson, and Narin 
1980; Hagstrom 1971). The addition of peer review data would also facilitate the 
acceptance by the scientific community of the findings from quantitative, bibliometric 
measures (Martin 1996). 

Nature of research activities 

The addition of more information about the research activities of each Nutrition 
department or program under study would increase the explanatory power of this study. 
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Of particular interested would be the inclusion of more data on research staffing, 
responsibilities, focus, and fimding. 

Research staffing and responsibilities 

It would be usefiil to know the number of researchers available in each 
department for each year of the study as well as the amount of time that each was 
expected to actually do research as opposed to other duties such as teaching, 
administration, and fimd-raising (Martin and Irvine 1983). The latter data would allow 
the calculation of a weight that could be multiplied by the number of researchers to 
normalize the staff size to reflect the actual number of researchers fimctioning as 
researchers (or FRs). For example, if Nutrition Department A has a designated research 
staff of 10, but requires an estimate average of 45% of their time be devoted to teaching 
and administrative duties, the weight would be calculated asl00% (total time) - 45% 
(average estimated amount of time doing non-research activities) = 65% (amount of time 
actually conducting research). So the number of FRs in Department A is 10 (size of 
research staff) x 0.65 (average time spent researching) or 6.5. The use of FRs would 
provide a more accurate assessment of the research capabilities of a given Nutrition 
department in terms of its research staffing. 
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Research focus 

Another aspect of research that would be interesting to know is the degree of 
focus and the changes in research subjects or topics over the time period of the study. Is 
a given department’s research efforts narrowly focused on a small array of research 
topics, or is the focus broad, ranging over a diverse number of topics? An intensive, 
narrow focus could possible account for the strong presence (and hence potential impact) 
of a given department’s research output in a given set of research fields and their 
associated journals. Conversely, a broader, more diverse research program may tend 
disperse the publication effort of a department, accounting for a relatively weak presence 
(and potential impact) in a larger set of research areas. 

Research funding 

It would also be useful to know the amount of research funding that each 
Nutrition department or program has available for each year from all sources (Martin and 
Irvine 1983). This would allow an evaluation of the relative influence the level of 
funding has on the relative research output and impact of a department or program. Such 
data could function as an indicator of how much research “bang” a given Nutrition 
department has produced for each research “buck” that it received from each of its 
funding sources. This in turn could perhaps serve as an indicator of their future research 
performance as well. 
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Non-citation based indicators of scientific achievement 

One of the indicators of research performance missing from this study is an 
evaluation of other non-citation based indicators of scientific achievement obtained by 
the researchers of a given Nutrition program for the period under study. These non- 
citation based indicators include esteem indicators such as medals, prizes (e.g.. Noble 
Prize), memberships in national academies (e.g.. National Academy of Science of the 
United States), academic rank, endowed chairs, and invitations to given prestigious 
lectures (Lawani and Beyer 1983; McAllister, Anderson, and Narin 1980; Hagstrom 
1971; Garfield 1989). It would be interesting to see if any of these non-citation based 
indicators of scientific achievement correlated with any of the bibliometric indicators 
used in this study. If so, then the non-bibliomentric indicators could serve to both 
provide not only additional qualitative information about the relative quality of a given 
department’s research output, but also provide a degree of validation for the results of the 
bibliometric evaluation as well. 

Departmentally determined peer programs 

Finally, it would also be useful to know the peer departments or programs that 
have been used previously for benchmarking purposes by each of the Nutrition programs 
under study. If none were used, it would still be useful to discover the ones that they 
would use if required to submit a list of peer programs for the university president, 
provost, or dean. Such a list may prove to be different than the one used by the university 
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for its evaluative purposes. It would provide a more accurate benchmarking tool since it 
originated within each Nutrition program or department, and based on their collective 
experience and understanding of both their department and the Nutrition field in general. 

Further Methodological Development 

The nature of the metrics or methodology of measuring research performance in 
this study could be further extended with the addition of citation mapping, testing of the 
indicators for the validity of the previously reported relationships with peer review and 
other non-bibliometric indicators of scientific achievement, as well as with size of 
research staff and amount of research funding. The stability or volatility of the patterns 
discovered in the findings of this study (see Figures 1-8) could be examined by extending 
the time frame from five to ten years. The methodology could also be extended by the 
use of a dynamic peer program set analogous to the dynamic journal set used by each 
Nutrition department under study to produce more accurate and relevant comparisons. 

Citation mapping 

Citation mapping is a method designed to graphically illustrate the underlying 
“structural and dynamic aspect of scientific research”(Noyons, Moed, and Luwel 1999, 

1 1 5). Although citation maps have been created using co-citation analysis, co-word 
analysis, or combinations of the two (Small 1999; White and McCain 1998; Noyons, 
Moed, and Luwel 1999), its use in this study would be restricted to co-citation analysis. 
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The data for co-citation analysis are “the number of times that selected author pairs are 
cited together” in the publications under consideration, and the goal is to “identify 
influential authors and display their interrelationships from the citation record”, or maps 
of'^writings related by use" [emphasis original] (White and McCain 1998, 327, 329), 
which for this study are scholarly, scientific journal publications. These maps show 
spatial relationships can “help facilitate our understanding of conceptual relationships and 
developments” within a scientific field (Small 1999, 799). Co-citation maps use “two 
dimensions to depict subject relationships,” under the assumption that “patterns of co- 
citation” define “the collective perceptions [of] citing authors” which in turn “create 
clusters of highly cited and co-cited works” (Small 1999, 800). By tracking the rate at 
which changes in these highly cited or co-cited articles occur, the rate of intellectual 
change in a given research field can be studied (Small 1999). Sudden changes are 
assumed to reflect revolutionary developments in a field. 

The use of co-citation mapping then can provide a visual display showing the 
relationship of a given Nutrition department’s publications to the research fronts or areas 
of “hottest” new or most popular research areas within the Nutrition field. In addition, it 
can show a given department’s researchers in relation to other researchers from other 
institutions, as well as show the nature of any interdisciplinary research across research 
areas within or without the Nutrition field. Co-citation mapping can also show the 
current concentrations of research specialties, and perhaps even help predict what areas 
of research will emerge as the new popular or “hot” research specialties (White and 




104 



McCain 1998; Small and Greenlee 1990; van Raan 1996). As an added bonus, the results 
of citation mapping can provide at least partial validation the findings from the evaluation 
of the departments using bibliometric indicators, while the latter findings can provide 
some validation of the structures revealed by the citation mapping (Noyons, Moed, and 
Luwel 1999). 

Relationships with other indicators 

The methodology could be further developed by using it as a vehicle for testing 
the relationship between the eight bibliometric indicators used in the study and other 
previously reported factors such as peer review results and other awards of scientific 
achievement, as well as with other research indicators such as size of research staff and 
amount and sources of funding. In the case of examining the eight bibliometric 
indicators’ relationship with peer review results and other awards of scientific 
achievement, the findings would either add to the growing both of literature providing 
empirical support, or provide some empirical grounds for questioning the relationships’ 
validity. The analysis of the bibliometric indicators relationship to the size of the 
research staff and amount of funding for Nutrition programs would serve to both gauge 
the relative importance of these two factors in assessing the research performance of 
Nutrition programs, an issue poorly covered in the existing literature, as well as there 
relative potential usefulness as core evaluative indicators. 
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Temporal scope 

Another way to develop the study’s methodology is to extend its temporal scope. 
For instance, the time period under study could be increased to cover ten years (1988- 
1997) by adding data from 1998 to 1991 and for 1997. The study’s timeframe cannot be 
moved any closer to 2001 due to a lack of available JCR data for 1998 on. The purpose 
of increasing the time period covered by the study would be to examine the degree of 
stability or volatility of the patterns discovered in the original study’s findings. This in 
turn would allow an exploration of the question of how large a time slice is necessary to 
get accurate and representative patterns for evaluation Nutrition departments. For 
example, if the patterns seem to be relatively stable, with relatively little variation 
between the five and ten year patterns, then it could be assumed that a five-year study 
period would be sufficient for evaluating the research performance of Nutrition programs. 
On the other hand, relatively volatile patterns showing a high degree of variation between 
the patterns show by ten-year and five-year studies would indicate the need for at least 
ten-year period would be required for accurately evaluating Nutrition departments. 

Dynamic peer program set 

It would be interesting to see if the development and use of a dynamic peer 
program or department set analogous to the dynamic journal set would prove to be a 
useful for improving the comparative accuracy of the ranking of the departments or 
programs under study. Such a dynamic peer program set, generated for each department 
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or program and for each year of the study would reflect the changes in the peer programs 
over time due to such factors as changes in a program’s primary mission (e.g., from 
research to teaching) or research (e.g., from applied to theoretical). These dynamic peer 
program sets would be determined at the department or program level by each 
department or program using their knowledge of their own programs and the Nutrition 
field as a whole or as revealed by citation mapping. Using a dynamic peer program set 
would also have the added advantage being the same level of aggregation as the study 
level, thereby increasing overall comparability of the peer departments. 

Summary 

In this study, a set of eight empirically derived and theoretically sound citation- 
based bibliometric indicators of scientific research performance were developed: P, 
FCSm, JCSm, C, CPP, CPP/FCSm, CPP/JCSm, and JCSm/FCSm. They were used in an 
exploratory study comparing the University of Tennessee-Knoxville’s Nutrition 
Department with three of its peer programs at the University of Florida, the University of 
Georgia, and Virginia Tech over a five-year period (1992-1996). The results of the study 
found that the UTK Nutrition Department ranked second in both the overall rankings and 
in the Actual Impact Group of indicators, and ranked third in the Publication Output and 
Benchmark Groups of indicators. 

Though limited in their ability to provide causal explanations, the bibliometric 
findings from this study can help the deans and department heads form the four programs 
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examined in the planning and management of their research units. In addition, they can 
use the bibliometric findings as augmentations to the traditional peer review evaluations 
of their unit’s research performance. 

The findings of this study can also provide an additional source of comparative 
data for further informetric and scientometric studies of departmental level research 
performance, of which there is little available in the literature. These findings can also 
provide further empirical data on the behavior of eight bibliometric indicators previously 
used in the literature, albeit modified for this study. 

Some of the subject areas touch upon in this study that merit further exploration 
are each program’s publication habits, its peer review ratings, the nature of its research 
activity, the inclusion of non-citation based indicators of scientific achievement for each 
program, and the incorporation of any data from previous used benchmarking programs. 
The methodology of this study could be further extended with the addition of citation 
mapping; validation of the previously reported relationships between the indicators and 
peer review; the addition of other non-bibliometric indicators of scientific achievement, 
the size of research staff, and amount of research funding; examining the stability or 
volatility of the patterns in the findings by extending the time frame from five to ten 
years; and exploring the use of a dynamic peer program set to produce more accurate and 
relevant comparisons. 
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APPENDIX A 

ANNOTATED BIBLIOGRAPHY OF SELECTED PAPAERS RELATING TO 
JOURNAL IMPACT FACTORS (JIFs) 



Campanario, J.M. 1996. The competition for journal space among referees, editors, 
and other authors and its influence on journals’ impact factors. Journal of the 
American Society for Information Science 47, no. 3 : 1 84- 1 92. 

Discovered a positive relationship between a journal’s JIF and its use by 
“journal- related authors” (e.g., external referees, or editors of that journal). 

Davis, G., and P. Royle. 1996. A comparison of Australian university output using 
journal impact factors. Scientometrics 35, no. 1; 45-58. 

Use of JIF as quality indicator allowed scaling (adjusting) results for 
institutional size. 

Garfield, E. 1989. Evaluating research: Do bibliometric indicators provide the best 
measures? In Essays of an Information Scientist: Vol. 12. Creativity, Delayed 
Recognition, and Other Essays. Philadelphia: Institute for Scientific Information 
Press, <http://www.garfield.library.upenn.edu/essays/vl2p093yl989.pdf>. 2 May 
2000 . 

Review of bibliometric performance measures “in terms of their strengths, 
weaknesses and particular applications.” Actually, this is an abridged version of 
Jean King’s 1987 article “A review of bibliometric and other science indicators 
and their role in research evaluation” {Journal of Information Science, 13, 261- 
276). 

Garfield, E. 1996. How can impact factors be improved? British Medical Journal 3X3: 
411-413. 

JIF misused as evaluative surrogates. It actually reflects a journal’s long 
term ability to attract the best articles (data fi"om medical journals). 

Garfield, E. 1997. Dispelling a few common myths about journal citation impacts. The 
Scientist 11 , no. 3: 11. <http://www.the-scientist.library.upenn.edu/ 
yrl997/feb/comm._970203.html>. 3 May 2000. 

Mainly a rebuttal of notion that journal size alone determines its impact 

factor. 

Garfield, E. 1998. The multiple meanings of impact factors. Journal of the American 
Society for Information Science 49, no. 8: 768. 

Point by point rebuttal of Harder and Nisonger’s criticism of JIF. 
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Garfield, E. 1998. The use of journal impact factors and citation analysis for 

evaluation of science. Paper presented at the 41^ Annual Meeting of the Council 
of Biology Editors, Salt Lake City, UT, 4 May 1998- 17 April 1998. <http:// 

WWW. garfield.upenn.edu/papers/eval_of_science_CBE(Utah).html>. 3 May 
2000 . 

Includes a discussion of using JIF as a surrogate for actual citation counts. 
In general he discourages such use, but discusses acceptable real life exception. 
Shows that long-term JIF favors established fields, while short-term (2-year) JIFs 
favor hot fields. Rebuttal of assertion that a field of study’s size has any 
noticeable effect on its journal’s JIFs. 

Garfield, E. 1999. Journal impact factor: A brief review. Canadian Medical Association 
Journal 161: 979-980. <http://www. cma.ca/cmaj/vol-161/issue-8/0979.htm 

Defense of JIF. JIF created as simple means of journal comparison 
regardless of size. Evolved in Europe to include both author and journal impact, 
which are two different things. Provides thoughts on correct use of JIF. 

Maintains that though JIF is not a perfect measure of quality, it fits well with the 
perception each field has of its journals. 

Harter, S.P., and T.E. Nisonger. 1997. ISI’s impact factor as misnomer: A proposed new 
measure to assess journal impact. Journal of the American Society for Information 
Science 4S, no. 12: 1146-1148. 

JIF is primarily a measure of a journal’s influence on the scholarly 
communication process, not just an indicator of journal productivity. The number 
of articles (size) of a journal distorts its relative impact. Smaller journals with 
fewer, higher quality articles have less impact on scholarly communications than 
large journals publishing more, lower quality articles. Recommends replacement 
of JIF with “article impact factor,” a new measure of journal influence. 

Hecht, F., B.K. Hecht, and A. A. Sandberg. 1998. The journal “impact factor”: A 

misnamed, misleading, misused measure. Cancer Genetics and Cytogenetics 104: 
77-81. 

JIFs misused as measures of relative importance of researchers, programs, 
and institutions. Recommends JIF be renamed “citation rate index” to prevent 
future abuse. 

Moed, H.F., and Th.N. van Leeuwen. 1995. Improving the accuracy of Institute for 

Scientific Information’s journal impact factors. Journal of the American Society 
for Information Science 46, no. 6: 461-467. 
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ISI’s JIFs inaccurate due to failure to clearly define document types. JIFs 
should be recalculated by document type. 

Peritz, B.C. (1995). On the association between journal circulation and impact factor. 
Journal ofinformation Science 21 , no. 1. 63-67. 

Discovered low correlation between a journal’s JIF and its circulation. 

Schoonbaert, D., and G. Roelants. 1996. Citation analysis for measuring the value of 
scientific publications: Quality assessment tool or comedy of errors.^ Tropical 
Medicine and International Health 1, no. 6: 739-752. 

Balanced, but generally positive assessment of using JIFs as quality 
measures in the sciences, as long as it’s limitations are understood. 

Seglen, P.O. 1992. The skewness of science. Journal of the American Society for 
Information Science 43, no. 9: 628-638. 

The variability of citation rates between articles in scientific journals is too 
great to make either JIF or article citation rates a viable measure of research 
productivity of individuals or groups. 

Seglen, P.O. 1994. Causal relationship between article citedness and journal impact. 
Journal of the American Society for Information Science 45, no. 1 : 1-11. 

Use of JIF to evaluate research units (“author, research group, institution, 
or country) can produce highly misleading results unless the size of the evaluated 
unit is equal in size to the world average for that unit. 

Seglen, P.O. 1997. Citations and journal impact factors. Questionable indicators of 
research quality. Allergy 52: 1050-1056.. 

Good review of criticisms of JIF and citation analysis, including the 
problems of citation motives, shortcomings of the ISI database, effect of a given 
research field on citation rates, English language bias. North American journal 
bias, effects of article length and review articles, and the problems with 
calculating JIFs. 

Sen, B.K. 1992. Documentation note: Normalized impact factor. Journal of 
Documentation 48, no. 3:318-325. 

Using a normalized JIF provides better indicator of comparative 
performance quality between labs doing diverse research. 
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Van Leeuwen, Th.N., H.F. Moed, and J. Reedijk. 1999. Critical comments on institute for 
scientific information journal impact factors; A sample of inorganic molecular 
chemistry journals. Journal of Information Science 25, no.6. 489-498. 

Problems of using the JIFs of science journals as quality measures: 
different document types, journal splitting into tow, journal name changes, rate of 
uncitedness of papers published, and the short length of citation time period (two 
years) upon which the JIF is calculated 

Vinkler, P. 1991. Possible causes of differences in information impact of journals from 
different subfields. Scientometrics 20, no. 1: 145-161. 

Using data from chemistry journals, found differences between JIFs due to 
journal size, “mean number of references per article,” information ageing rates, 
and subject subfields. Recommends use of a standardized JIF for comparisons 
between journals for difference subfields. 
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APPENDIX B 

EVALUATION OF BIBLIOMETRIC INDICATORS FROM THE NINE 
BIBLIOMTRIC STUDIES ANALYZED IN THE LITERATURE REVIEW FOR 

POSSIBLE USE IN THIS STUDY 



Evaluation Criteria 

The following criteria were used to evaluate the twenty-six bibliometric indicators 
used by the bibliometric studies reviewed for this study. In Table Bl, each criterion is 
listed followed by its possible values, and its definition. 



Table Bl. Selection criteria used to evaluate the bibliometric indicators used by the 
studies analyzed in the literature review. 



Criterion 


Possible Values 


Definition 


Definition 


Accept or reject 


What is the indicator? 


Calculation 


0 or 1 


How is the indicator created? 


Application 


0 or 1 


Does the indicator function as a primary or 
secondary indicator? 


Interpretation 


0 or 1 


What does the indicator mean? 


Peer Correlation 0 or 1 

Other 


Does the indicator correlate with the results 
of peer review or evaluation? (Name 
statistical test used, confidence interval, etc. 
if given). 


Correlation(s) 


0 or 1 


Do the results of the indicator correspond to 
one or more other indicators or methods of 
scientific evaluation (such as academic 
awards, membership in prestigious academic 
academies, etc). 
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Note(s): 

Values (accept or reject) : If the indicator it is not defined, or the definition is 
Table B1 (continued). 

not reasonably clear and understandable, then it is immediately rejected without further 
consideration. 

Values (0 or 1) : If the criteria is not defined or explained in the study (or studies) in 
which it is used, or done so poorly that the meaning is not clear, or the study results are 
conflicting, then it is given a value of 0 (zero). Otherwise, it is given a value of 1 (one). 
Primary indicators function mainly as a partial measure of a specified aspect (e.g., CPP 
functions as a primary indicator of publication impact). Optionally, it can be used for 
calculating other primary measures such CPP/FCSm and CPP/JCSm. 

Secondary indicators function mainly as the basis for calculating one or more other 
primary indicators (e.g., C is used mainly to calculate primary indicators such as CPP, 
CPPex, and CPP/FCSm). 



Evaluation Findings 

The following is an alphabetical list of all 26 bibliometric indicators encountered 
during the literature review of actual bibliometric studies evaluating the performance of 
academic departments. The rating for an Indicator is given in parentheses after its name. 
Source(s) lists reference(s) to the study in which the indicator was found. Any variation 
of an indicator (e.g., P and %P) is listed as separate indicator. Note that world-wide is 
often used to mean either other European countries, or other Western countries such as 
the United States, Japan, and Australia (van Raan & van Leeuwen, 1998), though it is not 
always possible to tell from the context in which it is used. Study group refers to the unit 
of analysis such as academic department, research center, or similar used in the source 
study (or studies). NA means that the information is not available or not given any of the 
Source(s). Journal set refers to a group of journals usually selected by the study group 
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as being their main publication outlet. Publications and papers are often used 
interchangeably. 
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Indicator: C (4) 

Deflnition; Number of citations (including self-citations) received for all the study 
group’s publications 
Calculation; Item count 
Application; Secondary indicator 

Interpretation; Indicator of impact or visibility (van Raan and van Leeuwen 1998) 

Peer Correlation; High; Spearman’s rank correlation (r*) at the 99% confidence level 
(Rinia et al. 1998); None with peer evaluation (Martin & and Irvine 1983); however, 
McAllister, Anderson, and Narin (1980) found a strong correlation (0. 7-0.9 product- 
moment and Spearman’s r). 

Other Correlation(s): None with highly cited papers, publications/researcher, or CPP 
(Martin and Irvine 1983); Correlation (0.5-0. 8, product-moment and Spearman’s r) with 
honorific awards, academic position, etc. (Myers 1970; McAllister, Anderson, and Narin 
1980) 

Source(s): 

Martin, B.R., and J. Irvine. 1983. Assessing basic research. Some partial indicators of 
scientific progress in radio astronomy. Research Policy 12; 61-90. 

McAllister, P.R., R.C. Anderson, and F. Narin. 1980. Comparison of peer and citation 
assessment of the influence of scientiflc journals. Journal of the American Society for 
Information Science (fAdiy): 147-152. 

Myers, C.R. 1970. Journal citations and scientific eminence in contemporary psychology. 
American Psychologist IS, T[0. 11; 1041-1048 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 

Comparative analysis of a set of bibliometric indicators and central peer review criteria; 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27; 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http;// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 
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Indicator: Cex (2) 

Deflnition; Number of citations (excluding self-citations) received for all the study 
group’s publications 

Calculation; (Total number of citations)-(total self-citations) 

Application; Secondary indicator 
Interpretation; NA 
Peer Correlation; NA 
Other Correlation(s): NA 
Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes; A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2 ; 115-131. 



Indicator: %Cex (1) 

Definition; A study group’s percentage of the number of citations (excluding self- 
citations) for all the study group’s publications 

Calculation; [(study group’s citation count)-(self-citations)/ (total of all study groups’ 

citations)-(self-citations) jlOO 

Application; NA 

Interpretation. NA 

Peer Correlation; NA 

Other Correlation(s): NA 

Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes; A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2; 115-131. 



Indicator; CPP (5) 

Definition; Average number of citations (including self-citations) per paper 
Calculation; C/P 
Application: Primary indicator 
Interpretation; Indicator of actual impact 

Peer Correlation; High; Spearman’s rank correlation (r*) at the 99% confidence level 
(Rinia et al. 1998); Also with peer evaluation, (Martin and Irvine 1983) 

Other Correlation(s): With highly cited papers and publications pier researcher (Martin 
and Irvine 1983). 

Source(s): 

Korevaar, J.C., and H.F. Moed. 1996. Validation of bibliometric indicators in the field of 
mathematics, 37, no. 1; 117-130. 
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Martin, B.R., and J. Irvine. 1983. Assessing basic research, Some partial indicators of 
scientific progress in radio astronomy. Research Policy 12. 61-90. 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 
Comparative analysis of a set ofbibliometric indicators and central peer review criteria; 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27: 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http:// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 



Indicator; CPPex (4) 

Definition; Average number of citations (excluding self-citations) per paper 

Calculation; Cex/P 

Application; Secondary indicator 

Interpretation: A partial measure of publication impact 

Peer Correlation; High; Spearman’s rank correlation (r*) at the 99% confidence level 
(Rinia et al. 1998) 

Other Correlation(s): NA 
Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes: A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2: 115-131. 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 
Comparative analysis of a set ofbibliometric indicators and central peer review criteria; 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27: 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http:// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 



Indicator; CPPex/Overall Mean (3) 

Definition; Ratio of citations per publication to average number of citations from all 
study groups 

Calculation; CPPex/Overall Mean 

Application; Primary indicator of the impact of a study group 

Interpretation: High impact > 1.2, Low impact < 0.8, otherwise is Average impact 

(between 0.8 and 1.2) 

Peer Correlation; NA 
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Other Correlation(s): NA 
Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes; A bibliometric study. Journal of the 
American Society for Information Science 50, no.2: 115-131. 



Indicator; CPP/FCSm (4) 

Deflnition; Citation rate of the study group, compared with the average citations of all 
the articles in a journal category for a field/sub-field as defined by yC/?’s journal 
categories. 

Calculation; CPP/FCSm 

Application; Primary indicator of a CPP normalized to the field-based citation average 
Interpretation; Impact of the study group’s research on its field or subfield. If the result 
is > 1.0, then the study group’s “work is cited more frequently than the field-based world 
average” (van Raan and van Leeuwen 1998) 

Peer Correlation: High; Spearman’s rank correlation (rg) at the 99% confidence level 
(Rinia et al. 1998). 

Other Correlation(s): NA 
Source(s): 

Korevaar, J.C., and H.F. Moed. 1996. Validation of bibliometric indicators in the field of 
mathematics. Scientometrics 37 , no. 1; 117-130. 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 

Comparative analysis of a set of bibliometric indicators and central peer review criteria; 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27; 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http;// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 



Indicator: CPP/JCSm (4) 

Definition: Citations per publication compared with the citation rate of the study group’s 
journal set; called a different name in Nederhof (1993) and Zachos (1991) 

Calculation; CPP/JCSm 

Application; Primary indicator of a CPP normalized to the journal-based citation average 
Interpretation; Impact of the study group’s research on its journal set. If it is > 1.0, then 
it is above “the journal-based world average” (van Raan and van Leeuwen 1998) 

Peer Correlation: High; Spearman’s rank correlation (r*) at the 99% confidence level 
(Rinia et al. 1998). 

Other Correlation(s): NA 
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Source(s): 

Nederhof, A.J., R.F. Meijer, H.F. Moed, and A.F.J. van Raan. 1993. Research 
performance indicators for university departments: A study of an agricultural university. 
Scientometrics 21 : 157-178. 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 
Comparative analysis of a set of bibliometric indicators and central peer review criteria: 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27: 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http:// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 

Zachos, G. 1991. Research output evaluation of two university departments in Greece 
with the use of bibliometric indicators. Scientometrics 21, no.2: 195-221. 

Indicator: CPR (2) 

Definition: Citation productivity per researcher within a research group 
Calculation: Number of citations/number of researchers 
Application: NA 

Interpretation: Estimate of the research impact or visibility of a research group 
Peer Correlation: NA 
Other Correlation(s): NA 
Source(s): 

Nederhof^ A.J., R.F. Meijer, H.F. Moed, and A.F.J. van Raan. 1993. Research 
performance indicators for university departments: A study of an agricultural university. 
Scientometrics 21 . 157-178. 

Zachos, G. 1991. Research output evaluation of two university departments in Greece 
with the use of bibliometric indicators. Scientometrics 21, no.2: 195-221. 



Indicator: FCSm (3) 

Definition: Mean citation rate of all the articles in the journal category for a field/sub- 
field as defined by JCR\ journal categories [mean Field Citation Score] 

Calculation: Total citations of the study group in their field or subfield/total papers in the 
study group’s field or subfield 
Application: Secondary indicator 

Interpretation: Provides a world-wide reference value or expected impact score for 
comparison (van Raan and van Leeuwen 1998) 

Peer Correlation: Low; Spearman’s rank correlation (r*) (Rinia et al. 1998) 

Other Correlation(s): NA 
Source(s): 
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Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 
Comparative analysis of a set ofbibliometric indicators and central peer review criteria: 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27: 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http.7/ 
sahara.fsw.leidenuniv.nl/cwts/rcivo a.html>. 28 June 2000. 



Indicator: HCP (4) 

Definition: Highly cited papers (or key papers) of a study group that received at least 15 
(or more) citations within the same one-year period. 

Calculation: Item count 

Application: Indicator of the number of high impact “discoveries” made by the study 
group 

Interpretation: Each of these key papers has made a very large impact in its time on the 
advancement of scientific knowledge. 

Peer Correlation: With peer evaluation 
Other Correlation(s): NA 
Source(s): 

Martin, B.R. 1996. The use of multiple indicators in the assessment of basic research. 
Scientometrics 36: 343-362. 

Martin, B.R., and J. Irvine. 1983. Assessing basic research. Some partial indicators of 
scientific progress in radio astronomy. Research Policy 12: 61-90. 



Indicator: JCSm (3) 

Definition: Average number of citations received by all articles in a specific journal 
during a given citation window (Korevaar and Moed 1996) [mean Journal Citation 
Score] 

Calculation: Total citations of study group in their Journal set/total papers in journal set 
Application: Secondary indicator 

Interpretation: Provides a world-wide reference value or expected journal impact score 
for comparison, allowing whether” the measured impact is above or below world 
average” (van Raan and van Leeuwen 1998) 

Peer Correlation: Low; Spearman’s rank correlation (r*) (Rinia et al. 1998) 

Other Correlation(s): NA 
Source(s): 

Korevaar, J.C., and H.F. Moed. 1996. Validation ofbibliometric indicators in the field of 
mathematics. Scientometrics 37, no. 1 : 117-130. 
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Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 
Comparative analysis of a set of bibliometric indicators and central peer review criteria: 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27: 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http:// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 



Indicator: JCSm/FCSm (4) 

Definition: Comparison of the impact of a journal to the world citation average of the 
field/sub-field (as defined by JC/?’s journal categories) to which the journal belongs 
(Korevaar and Moed 1996). 

Calculation: CPP/FCSm 

Application: Primary indicator of a journal’s impact on its field [a journal measure to 
replace ISI’s journal impact factor] (Korevaar and Moed 1996). 

Interpretation: If it is > 1.0, then the journal’s actual impact is above the world average 
(or expected impact); if = 1.0, then it is the same as the expected impact; if <1.0, then it is 
below the expected impact (Korevaar and Moed 1996). 

Peer Correlation: High; Spearman’s rank correlation (r*) at the 99% confidence level 
(Rinia et al. 1998). 

Other Correlation(s): NA 
Source(s): 

Korevaar, J.C., and H.F. Moed. 1996. Validation of bibliometric indicators in the field of 
mathematics. Scientometrics 31 , no. 1: 117-130. 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 

Comparative analysis of a set of bibliometric indicators and central peer review criteria: 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27: 95-107. 



Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http:// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 



Indicator: Overall Mean (2) 

Definition: Average number of citations from all study groups 

Calculation: Total number of citations/total number of publications 

Application: Secondary indicator 

Interpretation: NA 

Peer Correlation: NA 

Other Correlation(s): NA 
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Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes: A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2: 115-131. 



Indicator: P (3) 

Definition: Total number of journal articles, notes, and reviews published by the study 
group 

Calculation: Item count 
Application: Secondary indicator 

Interpretation: Indicator of scientific production (van Raan and van Leeuwen 1998), not 
of scientific progress (Martin and Irvine 1983) 

Peer Correlation: None (Rinia et al. 1998; Martin and Irvine 1983) 

Other Correlation(s): None with highly cited papers, publications/researcher, or CPP 
(Martin and Irvine 1983) 

Source(s): 

Martin, B.R., and J. Irvine. 1983. Assessing basic research. Some partial indicators of 
scientific progress in radio astronomy. Research Policy 12: 61-90. 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes. A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2: 115-131. 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and AF.J van Raan. 1998. 

Comparative analysis of a set of bibliometric indicators and central peer review criteria: 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27: 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http:// 
sahara.fsw.leidenuniv.nl/cwts/rcivo a.html>. 28 June 2000. 



Indicator: %P (1) 

Definition: A study group’s percentage of the total number of journal articles, notes, and 
reviews published by the all the study groups 

Calculation: (study group’s publications/total publications of all study groups)! 00 

Application: NA 

Interpretation: NA 

Peer Correlation: NA 

Other Correlation(s): NA 

Source(s): 
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Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes: A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2: 1 15-131. 
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Indicator; P/cit>90 (1) 

Deflnition; Number of publications for each study group among the 10% most frequently 
cited from all study groups” 

Calculation; Item count 
Application; NA 
Interpretation: NA 
Peer Correlation; NA 
Other Correlation(s): NA 
Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes; A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2. 115-131. 



Indicator; %P/cit>90 (1) 

Deflnition; Percentage of publications for each study group among the 10% most 
frequently cited from all study groups” 

Calculation; (P/cit>90 for a study group/P90)100 

Application; NA 

Interpretation: NA 

Peer Correlation; NA 

Other Correlation(s): NA 

Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes; A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2; 115-131. 



Indicator; P/cit>10 (1) 

Deflnition. Number of publications for each study group receiving more than 10 citations 

Calculation; Item count 

Application; NA 

Interpretation: NA 

Peer Correlation. NA 

Other Correlation(s): NA 

Source(s): 
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Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes: A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2 : 115-131. 



Indicator; %P/cit>10 (1) 

Definition; Percentage of publications for each study group receiving more than 10 
citations 

Calculation; (P/cit>10 for a study group/P/cit>10 for all study groups)100 

Application; NA 
Interpretation: NA 
Peer Correlation; NA 
Other Correlation(s): NA 
Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes: A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2: 115-131. 



Indicator; Pfte (2) 

Definition; Publication productivity per fte (or full-time equivalency) within a research 
group 

Calculation; Total publications/number of man-hours for research or fte 
Application: NA 

Interpretation: Estimate of scientific productivity of a research group 
Peer Correlation; NA 
Other Correlation(s): NA 
Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes: A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2: 115-131. 



Indicator; Pnc (1) 

Definition; Total count of publications not cited 

Calculation; Item count 

Application. NA 

Interpretation; NA 

Peer Correlation; NA 

Other Correlation(s): NA 

Source(s): 
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Indicator; %Pnc (1) 

Definition; Percentage of publications not cited 

Calculation; No. of uncited papers/total papers P)100 

Application; NA 

Interpretation; NA 

Peer Correlation; NA 

Other Correlation(s): NA 

Source(s): 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 
Comparative analysis of a set of bibliometric indicators and central peer review criteria: 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27: 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http:// 
sahara.fsw.leidenuniv.nl/cwts/rcivo_a.html>. 28 June 2000. 



Indicator. P90 (2) 

Definition; The 10% most frequently cited publications from all study groups (90*^ 
percentile) 

Calculation; Percentile 
Application; Secondary indicator 
Interpretation: NA 
Peer Correlation; NA 
Other Correlation(s): NA 
Source(s): 

Noyons, E.C.M., H.F. Moed, and M. Luwel. 1999. Combining mapping and citation 
analysis for evaluative bibliometric purposes: A bibliometric study. Journal of the 
American Society for Information Science 50, no. 2 : 115-131. 



Indicator; PPR (4) 

Definition; Publication productivity per researcher within a research group 
Calculation; Total publications/number of researchers 
Application: NA 

Interpretation: Estimate of scientific productivity of a research group 
Peer Correlation; With peer evaluation (Martin and Irvine, 1983) 
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Other Correlation(s): With highly cited papers and CPP (Martin and Irvine 1983). 
Source(s): 

Martin, B.R., and J. Irvine. 1983. Assessing basic research, Some partial indicators of 
scientific progress in radio astronomy. Research Policy 12; 61-90. 

Zachos, G. 1991. Research output evaluation of two university departments in Greece 
with the use of bibliometric indicators. Scientometrics 21, no.2: 195-221. 



Indicator; %Selfcite (3) 

Definition: Percentage of papers not cited 
Calculation; (Total self-cite^total cites C) 100 
Application; NA 

Interpretation: An indication either of a study group’s relative isolation due to its focus 
on currently unfashionable research topics, or as an indication of the uniqueness/newness 
of its research focus. 

Peer Correlations; High/negative (i.e., lower self-cite percentages correspond to higher 
peer ratings); Spearman’s rank correlation (rs) at the 99% confidence level (Rinia et al. 
1998). 

Other Correlation(s): NA 
Source(s): 

Rinia, E.J., Th.N. van Leeuwen, H.G. van Vuren, and A.F.J van Raan. 1998. 

Comparative analysis of a set of bibliometric indicators and central peer review criteria: 
Evaluation of condensed matter physics in the Netherlands. Research Policy 27; 95-107. 

Van Raan, A.F.J., and Th.N. van Leeuwen. 1998. Assessment of multidisciplinary, 
applied research using bibliometric methods in nutrition and food research. <http;// 
sahara.fsw.leidenuniv.nl/cwts/rcivo a.html>. 28 June 2000. 



Evaluation Results 

The selection of the actual indicators from those listed above was ultimately a 
subjective judgement based on a particular indicator’s rating, type (primary or 
secondary), and if it required the exclusion of self-citations in its calculation. Since it 
was previously decided that it was unnecessary to remove all self-citations from the data 
before use, all indicators requiring it (e.g., CPPex and Cex) were eliminated from further 
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consideration. All remaining indicators with ratings of 1 or 2 were rejected for having 
too few desirable criteria to merit further consideration. All the remaining indicators with 
ratings of 4 and 5 that were classed as “primary indicators” were thought the most useful, 
since each included at least one correlation criterion. The remaining indicators rated with 
rating and classed as “secondary indicators” were retained, with the balance 
The selection results are summarized in Table B2. 



at least a 3 
discarded. 
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Table B2. Selected bibliometric indicators, by type and rating order. 



Type 


Rating 


Indicator 


Primary 


5 


CPP (Average citations per paper) 




4 


C (Total citations) 




4 


CPP/FCSm (Citations per paper compared with the 
expected field’s citation rate) 




4 


CPP/JCSm (Citations per paper compared with expected 
journal set citation rate) 




4 


JCSm/FCSm (Comparison of the impact of a journal to the 
world citation average of the field/sub-field) 




4 


HCP (Highly cited papers, or key papers) 




4 


PPR (Publications per researcher) 


Secondary 


3 


FCSm (Average citation rate of the field or sub-field) 




3 


JCSm (Average citation rate of the journal set) 




3 


P (Total publications) 
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APPENDIX C 

EVALUATION OF THE NUTRITION PROGRAMS FROM EACH UTK PEER 
INSTITUTION FOR USE IN THIS STUDY 

Evaluation Criteria 

The following criteria were used to evaluate and rate the relative comparability of 
each peer institution’s Nutrition program with UTK’s program (see Table Cl). The 
higher a program or department’s rating, the greater is its similarity to UTK’s Nutrition 
Department. Data used for the evaluation was taken from the website of each institution, 
the URL of which was active as of 4 October 2000. 



O 
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Table Cl. Criteria for evaluating Nutrition departments or programs from UTK peer 
institutions for inclusion in this study. 



Criterion Possible Values 

Program Accept or reject 

Program Type 

Disciplinary 1 

Interdisciplinary 0 
Program Focus 

Traditional 1 

Non-traditional 0 

Faculty Size/Ratio Peer/UTK 

Faculty Type 

Research 0 

Research/extension 1 

Graduate Program 



None 0 

Masters 1 

Doctoral 1 

Active Research 0 or 1 



Definition 

Does the peer institution have a Nutrition 
Department or equivalent program? 

What is the nature of the department or 
program? 

Traditional, discipline specific academic 
department. 

Joint venture of many departments from 
different disciplines. 

What is the department or program’s 
main teaching and research subject focus? 
Subjects such as foods, nutrition, 
and biochemistry. 

Subjects such as nutrition and public health, 
nutrition and medicine, or human and 
animal/avian nutrition. 

How many faculty in the department or 
program? How does it compare in size to 
UTK faculty? 

What is the primary responsibility of the 
department or program’s faculty? 

Mainly to conduct research. 

To conduct both research and extension 
work. 

Does it have a graduate program in 
Nutrition? 



Does it have an active research component? 



Note(s): 

Values (accept or reject) : If the peer institution does not indicate the presence of a 
Nutrition Department or equivalent program on its website, then it is immediately 
rejected from further consideration. 

Values (0 or 1) : If the department does not have a graduate program, or has an 
interdisciplinary program or has a non-traditional research and teaching focus, or has 




xOU 



138 



only research faculty, or if it is not a Masters or Doctorate in Nutrition Science degree 
(e.g., certification program, internship, degree in Dietetic Administration, Hospitality, 
Health Education, etc.); or if there is no indication on the website that the department is 

Table Cl. (continued) 

actively engaged in nutrition related research, such as a list of current research projects, 
grants, etc., then it is given a value of 0 (zero). Otherwise, it is given a value of 1 (one). 
Peer/UTK: A comparative value calculated by dividing the number of faculty in the peer 
institution’s department or program by the number of faculty in the UTK Nutrition 
Department (i.e., 17 ). The resulting number is subtracted from the 1, and the absolute 
value of that result is in turn subtracted from rating. 

Active Research Component refers to a list of current or very recently completed research 
projects, not just to a list of research interests given on the department or program’s 
website. 

Equivalent programs refers to alternate names for departments that focus on nutrition as 
their academic discipline, such as Nutrition Sciences, Human Nutrition, Department of 
Foods and Nutrition, but excluding Food Science and Technology, Food Service, and 
similar applied science and technology programs. 
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Evaluation Findings 

The following is an alphabetical list of the peer institutions for the University of 

Tennessee-Knoxville with the results of the evaluation. Note that the total rating for each 

peer institution is given in parentheses after its name. 

Peer Institution; University of Florida (5.12) 

Program; Department of Food Science and Human Nutrition 

Program Type; Disciplinary 

Program Focus; Traditional 

Faculty Size/Ratio; 32/1.88 

Faculty Type; Research and extension 

Graduate program; MS and PhD 

Active research; List of research projects with active faculty investigators 
Source(s); http;//fshn. ifas.ufl.edu/academic.htm; http.V/fshn. ifas.ufl.edu/research.htm; 
http ;//fshn. ifas. ufl . edu/faculty . htm 



Peer Institution; University of Georgia (5.59) 

Program; Department of Foods and Nutrition 

Program Type. Disciplinary 

Program Focus; Traditional 

Faculty Size/Ratio; 24/1.41 

Faculty Type; Research and extension 

Graduate program; MS and PhD 

Active research; List of current research projects 

Source(s); http;//www.fcs.uga.edu/fdn/graduate; 

http;//www.fcs.uga.edu/fdn/people/index.php3?FDN+faculty 



Peer Institution; University of Kentucky (2.59) 

Program; Department of Nutrition and Food Science 

Program Type; Disciplinary 

Program Focus. Traditional 

Faculty Size/Ratio; 10/0.59 

Faculty Type. Research 

Graduate program; MS in Hospitality and Dietetic Administration 
Active research; Two current projects 
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Source(s); http.7/www.uky.edu/HES/NFS/nutritiongrad-new-hosp.html; 
http ://www. rgs.uky. edu/c^digest/index/html; 
http://www.uky.edu/HES/research%20bric/faculty.htm 



140 



Peer Institution; University of Maryland at College Park (2.82) 

Program; Nutrition Program, Department of Nutrition and Food Science 

Program Type; Interdisciplinary 

Program Focus; Non-traditional 

Faculty Size/Ratio; 20/1 .18 

Faculty Type: Research 

Graduate program; MS and PhD 

Active research; Requires completion of research project by graduate students 
Source(s); http://www.gradschool.umd.edu/catalo^programs/nutr.htm; 
http.V/www.agnr.umd.edu/users/nfsc/gradnutr.htm 



Peer Institution; University of North Carolina at Chapel Hill (1 .06) 

Program; Department of Nutrition 

Program Type: Interdisciplinary 

Program Focus; Non-traditional 

Faculty Size/Ratio. 50/2.94 

Faculty Type: Research 

Graduate program. MS and PhD 

Active research; List of ongoing research 

Source(s): http;//www.sph.unc.edu/nutr/Degree/degree.html; 

http://www.sph.unc.edu/nutr/Research/research.html; 

http://www.sph.unc.edu/nutr/Divisions/biochem.html 



Peer Institution; University of Oklahoma (3.41) 

Program; Department of Nutritional Sciences 

Program Type. Disciplinary 

Program Focus; Traditional 

Faculty Size/Ratio; 7/0.41 

Faculty Type: Research 

Graduate program; MS 

Active research; List of research activities 

Source(s); http://rentsvl .ouhsc.edu/ahealth/ns.htm; 

http://rentsvl .ouhsc.edu/ahealth/research.htm; http.V/rentsvl .ouhsc.edu/ahealth/fans.htm 
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Peer Institution; University of South Carolina (0) 

Program; None to date, but one is pending 
■ Program Type; NA 
Program Focus; NA 
Faculty Size/Ratio; NA 
Faculty Type; NA 
Graduate program; None 
Active research; None 

Source(s): http.V/www.sph.sc.edu/sphdept.htm; http;//www. sph.sc.edu/news/nutrctr.htm 



Peer Institution; University of Texas at Austin (3.94) 

Program; Department of Nutritional Sciences 

Program Type; Disciplinary 

Program Focus; Traditional 

Faculty Size/Ratio; 16/0.94 

Faculty Type; Research 

Graduate program; MA and PhD 

Active research; List of faculty research interests 

Source(s): http ;//www. utexas.edu/depts/he/nutr_science/; 

http ; //www . utexas . edu/dept s/he/nutr_science/facu Ity ; 

http;//www.utexas.edu/depts/he/nutr_science/faculty/grfaculty 



Peer Institution; University of Virginia (0) 

Program; None 

Program Type. NA 

Program Focus; NA 

Faculty Size/Ratio; NA 

Faculty Type; NA 

Graduate program; None 

Active research; None 

Source(s): http;//www. Virginia.edu/depts.html 



Peer Institution; Virginia Polytechnic Institute and State University (5.59) 

Program; Department of Human Nutrition, Foods, and Exercise 

Program Type; Disciplinary 

Program Focus; Traditional 

Faculty Size/Ratio; 24/1.41 

Faculty Type; Research and Extension 

Graduate program. MS and PhD 
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Active research; Molecular Nutrition Lab 
Source(s) : http ://www. chre.vt.edu/Admin/HNFE; 
http://www.chre.vt.edu/Admin/HNFE/fac/Davis/current_research.htm; 
http://www.chre.vt.edu/jhwms/hnfegrad/faculty.html 



Evaluation Results 

The following table lists in rating order from highest to lowest the peer 
institutions that qualify for possible use as sources for comparison in this study (see Table 
C2). Note that peer institutions with the same rating are listed alphabetically. 



Table C2. UTK peer institutions in rating order, by institution and department (or 
program) name. 



Rank 


Peer Institution 


Department (or Program) 


5.59 


University of Georgia 


Department of Foods and Nutrition 


5.59 


Virginia Polytechnic 
Institute and State 
University 


Department of Human Nutrition, Foods, and 
Exercise 


5.12 


University of Florida 


Department of Food Science and Human Nutrition 


3.94 


University of Texas at 
Austin 


Department of Nutritional Sciences 


3.41 


University of Oklahoma 


Department of Nutritional Sciences 


2.82 


University of Maryland at 
College Park 


Nutrition Program, Department of Nutrition and 
Food Science 


2.59 


University of Kentucky 


Department of Nutrition and Food Science 


1.06 


University of North 
Carolina at Chapel Hill 


Department of Nutrition 
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APPENDIX D 



DATA WORKSHEET FORMS 



Figure Dl. Journal Data Worksheet Form 



Journal Data Worksheet for (project): 


Institution: 


Year: 


Source(s): 


Journal 


ISI Field 


ISI JLF 
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C 
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CPP/JCSm 
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Figure D2. Field Data Worksheet Form 



Field Data Worksheet for (project): 


Institution: 


Year: 


Source(s): 


ISI Field 


Number of Journals 


Total JIF 
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APPENDIX E 

JOURNAL SET DATA FOR EACH NUTRITION PROGRAM, BY YEAR, 1992-1996 



1992 Journal Set Data 

Table El. Journal set data for the Department of Food Science and Human Nutrition, 
University of Florida, 1992. 



Journal Title 


Field Name 


ISI JIF 


P 


Clinical Research 


Medicine, Research & 
Experimental 


57.778 


1 


FASEB Journal 


Biochemistry & Molecular 
Biology 


15.115 


8 


Abstracts of Papers of the 
American Chemical Society 


Chemistry 


8.000 


5 


Hepatology 


Gastroenterology & 
Hepatology 


5.569 


1 


Molecular Microbiology 


Biochemistry & Molecular 
Biology 


5.142 


1 


Biochemical Journal 


Biochemistry & Molecular 
Biology 


4.262 


1 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


2.864 


1 


Journal of Pediatrics 


Pediatrics 


2.609 


1 


Toxicology & Applied 
Pharmacology 


Pharmacology & Pharmacy 


'2.490 


1 


Journal of Laboratory & 
Clinical Medicine 


Medicine, General & 
Internal 


2.244 


1 


Journal of Nutrition 


Nutrition & Dietetics 


1.977 


6 


Enzyme & Microbial 
Technology 


Biotechnology & Applied 
Microbiology 


1.784 


1 


Proceedings of the Society for 
Experimental Biology& 
Medicine 


Medicine, Research & 
Experimental 


1.675 


1 


Critical Reviews in Food 
Science & Technology 


Nutrition & Dietetics 


1.377 


1 


Journal of Agricultural & Food 
Chemistry 


Food Science & 
Technology 


1.342 


1 


Journal of Animal Science 


Agriculture, Dairy & 
Animal Science 


1.335 


1 
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Table El. (continued) 



Journal Title 


Field Name 


ISl JIF 


P 


Journal of Food Protection 


Food Science & 
Technology 


1.292 


1 


International Journal of 
Immunopharmacology 


Pharmacology & Pharmacy 


1.170 


1 


Toxicology Letters 


Toxicology 


1.112 


1 


Journal of the American 
Dietetic Association 


Nutrition & Dietetics 


1.090 


1 


Journal of Food Safety 


Food Science & 
Technology 


1.047 


1 


Journal of Food Science 


Food Science & 
Technology 


0.872 


4 


Comparative Biochemical 
Physiology B- Biochemistry 


Biochemistry & Molecular 
Biology 


0.685 


2 




Total: 


122.831 


43 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISl JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1 994 Journal Citation Report ( 1 995). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Food Science and Human Nutrition, University of Florida. Data is for the year 
1992, and is from the Web of Science (2000), accessed 8 October 2000. 

Source (s): 

Institute for Scientific Information. 1995 . 1994 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISl Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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Table E2. Journal set data for the Department of Foods and Nutrition, University of 
Georgia, 1992. 



Journal Title 


Field Name 


ISIJIF 


P 


FASEB Journal 


Biocehmistry & Molecular 
Biology 


15.115 


7 


American Journal of 
Physiology 


Physiology 


3.276 


1 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


2.864 


1 


Journal of Nutrition 


Nutrition & Dietetics 


1.977 


3 


Journal of Applied Physiology 


Physiology 


1.852 


1 


Brain Research Bulletin 


Neurosciences 


1.811 


1 


International Joumal of 
Obesity 


Nutrition & Dietetics 


1.568 


1 


Pharmacology Biochemistry & 
Behavior 


Pharmacology & Pharmacy 


1.450 


1 


Critical Reviews in Food 
Science & Nutrition 


Food Science & 
Technology 


1.377 


1 


Physiology & Behavior 


Physiology 


1.110 


2 


Journal of Nutritional 
Biochemistry 


Nutrition & Dietetics 


1.015 


1 


Biochemical Archives 


Biochemistry & Molecular 
Biology 


0.714 


1 




Total: 


34.129 


21 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1994 Journal Citation Report (1995). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Foods and Nutrition, University of Georgia. Data is for the year 1992, and is from 
the Web of Science (2000), accessed 8 October 2000. 

Source (s): 

Institute for Scientific Information. 1995 . 1994 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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Table E3. Journal set data for the Department of Nutrition, University of Tennessee- 
Knoxville, 1992. 



Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


15.115 


3 


American Journal of 
Physiology 


Physiology 


3.276 


1 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


2.864 


1 


Diabetes Care 


Endocrinology & 
Metabolism 


2.755 


1 


Journal of Nutrition 


Nutrition & Dietetics 


1.977 


1 


American Journal of 
Hypertension 


Cardiovascular System 


1.907 


1 


Lipids 


Nutrition & Dietetics 


1.530 


1 


Journal of the American 
Dietetic Association 


Nutrition & Dietetics 


1.090 


5 


Journal of Food Science 


Food Science & 
Technology 


0.872 


1 


Biochemical Archives 


Biochemistry & Molecular 
Biology 


0.714 


1 


Journal of the Canadian 
Dietetic Association 


Nutrition & Dietetics 


0.220 


1 




Total: 


32.32 


17 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISl JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1994 Journal Citation Report (1995). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Nutrition, University of Tennessee- Knoxville. Data is for the year 1992, and is 
from the Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1995. 1994 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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Table E4. Journal set data for the Department of Human Nutrition, Foods, and Exercise, 
Virginia Polytechnic Institute and State University, 1992. 



Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


15.115 


1 


JAMA-Joumal of the 
American Medical Association 


Medicine, General & 
Internal 


6.863 


1 


Journal of Agricultural & Food 
Chemistry 


Food Science & 
Technology 


1.342 


1 


Journal of Animal Science 


Agriculture, Dairy & 
Animal Science 


1.335 


1 


Journal of the American 
Dietetic Association 


Nutrition & Dietetics 


1.090 


2 


Journal of Nutritional 
Education 


Nutrition & Dietetics 


0.309 


1 


Journal of Food Quality 


Food Science & 
Technology 


0.219 


1 


Ecology of Food & Nutrition 


Nutrition & Dietetics 


0.211 


1 




Total: 


26.484 


9 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1994 Journal Citation Report {\995). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Human Nutrition, Foods, and Exercise, Virginia Polytechnic Institute and State 
University. Data is for the year 1992, and is from the Web of Science (2000), accessed 8 October 
2000. 

Source(s): 

Institute for Scientific Information. 1995 . 1994 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database., Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia; Institute for Scientific Information. 
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1993 Journal Set Data 

Table E5. Journal set data for the Department of Food Science and Human Nutrition, 
University ofFlorida, 1993. 



Journal Title 


Field Name 


ISI JIF 


P 


Abstracts of Papers of the 
American Chemical Society 


Chemistry 


31.000 


1 


FASEB Journal 


Biochemistry & Molecular 
Biology 


13.404 


7 


Proceedings of the National 
Academy of Sciences of the 
USA 


Multidisciplinary Studies 


10.520 


1 


American Journal of 
Physiology 


Physiology 


3.244 


1 


Journal of Nutrition 


Nutrition & Dietetics 


1.925 


6 


Proceedings of the Society for 
Experimental Biology & 
Medicine 


Medicine, Research and 
Experimental 


1.654 


1 


Nutritional Reviews 


Nutrition & Dietetics 


1.457 


1 


Journal of Agricultural & Food 
Chemistry 


Food Science & 
Technology 


1.434 


1 


Journal of the American Oil 
Chemists Society 


Food Science & 
Technology 


1.228 


3 


Journal of the American 
Dietetic Association 


Nutrition & Dietetics 


1.202 


2 


Journal of Food Science 


Food Science & 
Technology 


0.998 


5 


Current Microbiology 


Microbiology 


0.962 


2 


Food Technology 


Food Science & 
Technology 


0.857 


1 


Journal of Pharmacy & 
Pharmacology 


Pharmacology & Pharmacy 


0.848 


1 


Journal of the Science of Food 
& Agriculture 


Food Science & 
Technology 


0.842 


1 


Letters in Applied 
Microbiology 


Biotechnology & Applied 
Microbiology 


0.764 


1 


ACS Symposium Series 


Chemistry 


0.655 


2 


Biological Trace Element 
Research 


Biochemistry & Molecular 
Biology 


0.471 


1 




Total: 


73.465 


38 
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Table E5. (continued) 

Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISl JIF is the journal impact factor for the journal as foimd in the Institute for Scientific 
Information’s 1995 Journal Citation Report (1996). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Food Science and Human Nutrition, University of Florida. Data is for 1993, and 
is from the Web of Science (2000), accessed 8 October 2000. 

Source (s): 

Institute for Scientific Information. 1996. 1995 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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Table E6. Journal set data for the Department of Foods and Nutrition, University of 
Georgia, 1993. 
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Journal Title 


Field Name 


ISI JIF 


P 


Abstracts of Papers of the 
American Chemical Society 


Chemistry 


31.000 


I 


FASAB Journal 


Biochemistry & Molecular 
Biology 


13.404 


17 


Endocrinology 


Endocrinology & 
Metabolism 


4.736 


I 


Biotechniques 


Biomethods 


2.305 


I 


Journal of Nutrition 


Nutrition & Dietetics 


1.925 


3 


Society of Experimental 
Biology and Medicine 


Medicine, Research & 
Experimental 


1.654 


I 


Journal of Nutritional 
Biochemistry 


Nutrition & Dietetics 


I.I88 


2 


Physiology & Behavior 


Physiology 


I.I7I 


I 


Progress in Food & Nutritional 
Science 


Nutrition & Dietetics 


I.OOO 


I 


Journal of Pharmacological & 
Toxicological Methods 


Pharmacology & Pharmacy 


0.758 


I 


Hormone & Metabolic 
Research 


Endocrinology & 
Metabolism 


0.674 


I 


Nutrition 


Nutrition & Dietetics 


0.656 


I 


Biology of the Neonate 


Pediatrics 


0.634 


I 


Nutritional Research 


Nutrition & Dietetics 


0.559 


2 


Comparative Biochemical 
Physiology A- Physiology 


Physiology 


0.531 


I 


Biological Trace Element 
Research 


Biochemistry & Molecular 
Biology 


0.471 


I 


Biochemical Archives 


Biochemistry & Molecular 
Biology 


0.434 


I 




Total: 


63.100 


37 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1 995 Journal Citation Report ( 1 996). 
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Table E6. (continued) 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Foods and Nutrition, University of Georgia. Data is for 1993, and is from the Web 
of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1996. 1995 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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Table E7. Journal set data for the Department of Nutrition, University of Tennessee- 
Knoxville, 1993. 
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Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


13.404 


6 


Hypertension 


Cardiovascular Systems 


4.981 


2 


Archives of Internal Medicine 


Medicine, General & 
Internal 


4.166 


I 


Pediatrics 


Pediatrics 


2.710 


I 


Journal of Nutrition 


Nutrition & Dietetics 


1.925 


2 


American Journal of 
Hypertension 


Cardiovascular Systems 


1.880 


I 


Experimental Gerontology 


Geriatrics & Gerontology 


1. 124 


I 


Journal of the American 
College of Nutrition 


Nutrition & Dietetics 


0.925 


I 


Nutrition Research 


Nutrition & Dietetics 


0.559 


I 


Biochemical Archives 


Biochemistry & Molecular 
Biology 


0.434 


I 


Journal of Food Quality 


Food Science & 
Technology 


0.244 


I 




Total: 


32.352 


18 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1995 Journal Citation Report (1996). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Nutrition, University of Tennessee- Knoxville. Data is for 1993, and is from the 
Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1996 . 1995 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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Table E8. Journal set data for the Department of Human Nutrition, Foods, and Exercise, 
Virginia Polytechnic Institute and State University, 1993. 



Journal Title 


Field Name 


ISI JIF 


P 


Journal of Chromatography 


Chemistry 


2.296 


1 


Journal of Nutrition 


Nutrition & Dietetics 


1.925 


1 


Journal of Cereal Science 


Food Science & 
Technology 


1.373 


1 


Journal of the American 
Dietetic Association 


Nutrition & Dietetics 


1.202 


1 


International Journal of Sports 
Medicine 


Sports Sciences 


0.810 


1 


Journal of the Society of Dairy 
Technology 


Food Science & 
Technology 


0.250 


1 


Plant Foods for Human 
Nutrition 


Food Science & 
Technology 


0.117 


1 




Total: 


7.973 


7 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1995 Journal Citation Report (1996). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Human Nutrition, Foods, and Exercise, Virginia Polytechnic Institute and State 
University. Data is for 1993, and is fi'om the Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1996. 1995 JCR: Journal Citation Reports, Science Edition, A 
Bib Home trie Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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1994 Journal Set Data 



Table E9. Journal set data for the Department of Food Science and Human Nutrition, 
University of Florida, 1994. 



Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


13.771 


14 


JAMA- Journal of the 
American Medical Association 


Medicine, General & 
Internal 


9.277 


1 


Annual Review of Nutrition 


Nutrition & Dietetics 


5.867 


1 


Biochemistry Journal 


Biochemistry & Molecular 
Biology 


3.687 


2 


Journal of Acquired Immune 
Deficiency Syndromes & 
Human Retrovirology 


Immunology 


3.511 


1 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


3.245 


1 


Applied & Environmental 
Microbiology 


Biotechnology & Applied 
Microbiology 


3.129 


1 


Vaccine 


Immunology 


2.323 


1 


Journal of Nutrition 


Nutrition & Dietetics 


1.986 


2 


Metabolism- Clinical & 
Experimental 


Endocrinology & 
Metabolism 


1.788 


1 


FEMS Microbiology Letters 


Microbiology 


1.735 


1 


Journal of Agricultural & Food 
Chemistry 


Food Science & 
Technology 


1.732 


2 


Journal of Food Science 


Food Science & 
Technology 


1.225 


1 


Journal of Nutritional 
Biochemistry 


Nutrition & Dietetics 


1.172 


2 


Journal of the American 
Dietetic Association 


Nutrition & Dietetics 


1.158 


1 


Current Microbiology 


Microbiology 


1.092 


1 


American Journal of Enology 
& Viticulture 


Food Science & 
Technology 


0.908 


1 


ACS Symposium Series 


Chemistry 


0.657 


1 


Nutritional Research 


Nutrition & Dietetics 


0.638 


1 



O 
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Table E9. (continued) 



Journal Title 


Field Name 


ISI JIF 


P 


Biological Trace Element 
Research 


Biochemistry & Molecular 
Biology 


0.627 


1 


Food Research International 


Food Science & 
Technology 


0.613 


1 


Medical Hypotheses 


Medicine, Research & 
Experimental 


0.561 


1 


Journal of Food Quality 


Food Science & 
Technology 


0.351 


1 




Total: 


61.053 


40 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1996 Journal Citation Report (1997). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Food Science and Human Nutrition, University of Florida. Data is for 1994, and 
is from the Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1997. 1996 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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Table ElO. Journal set data for the Department of Foods and Nutrition, University of 
Georgia, 1994. 



Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


13.771 


13 


Journal of Clinical 
Investigation 


Medicine, Research & 
Experimental 


9.486 


1 


Journal for the American 
Society of Nephrology 


Urology & Nephrology 


6.846 


1 


Journal of Cellular 
Biochemistry 


Cell Biology 


3.471 


2 


American Journal of 
Physiology 


Physiology 


3.323 


2 


Journal of Nutrition 


Nutrition & Dietetics 


1.986 


2 


Proceedings of the Society for 
Experimental Biology & 
Medicine 


Medicine, Research & 
Experimental 


1.827 


1 


Physiology & Behavior 


Physiology 


1.242 


1 


Journal of Nutritional 
Biochemistry 


Nutrition & Dietetics 


1.172 


\\ 


Food Technology 


Food Science & 
Technology 


1.100 


1 


International Journal for 
Vitamin & Nutrition Research 


Nutrition & Dietetics 


0.764 


1 


Amino Acids 


Biochemistry & Molecular 
Biology 


0.744 


1 


Biochemical Archives 


Biochemistry & Molecular 
Biology 


0.373 


2 




Total: 


46.105 


29 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISl JIF is the journal impact factor for the journal as fiaimd in the Institute for Scientific 
Information’s 1 996 Journal Citation Report ( 1 997). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Foods and Nutrition, University of Georgia. Data is far 1994, and is from the Web 
of Science (2000), accessed 8 October 2000. 
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Table ElO. (continued) 

Sourcefs): 

Institute for Scientific Information. 1997. 1996 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia; Institute for Scientific Information. 
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Table El 1 . Journal set data for the Department of Nutrition, University of Tennessee- 
Knoxville, 1994. 



Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


13.771 


10 


Journal of Lipid Research 


Biochemistry & Molecular 
Biology 


3.620 


I 


American Journal of 
Physiology 


Physiology 


3.323 


I 


Biochemical & Biophysical 
Research Communications 


Biochemistry & Molecular 
Biology 


2.872 


I 


Journal of Nutrition 


Nutrition & Dietetics 


1.986 


2 


Journal of the American 
College of Nutrition 


Nutrition & Dietetics 


1.226 


2 


Comparative Biochemistry & 
Physiology A- Physiology 


Biochemistry & Molecular 
Biology 


0.618 


I 




Total: 


27.416 


18 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1996 Journal Citation Report {1991). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Nutrition, University of Tennessee- Knoxville. Data is for 1994, and is from the 
Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1997. 1996 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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Table E12. Journal set data for the Department of Human Nutrition, Foods, and Exercise, 
Virginia Polytechnic Institute and State University, 1994. 



Journal Title 


Field Name 


ISI JIF 


P 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


3.245 


1 


Journal of Nutritional 
Biochemistry 


Nutrition & Dietetics 


1.172 


1 


Environmental Health 
Perspectives 


Environmental Sciences 


1.688 


1 


Journal of Dairy Science 


Food Science & 
Technology 


1.139 


1 


Nutritional Research 


Nutrition & Dietetics 


0.638 


1 


Journal of Nutritional 
Education 


Nutrition & Dietetics 


0.575 


1 


Journal of the Society of Dairy 
Technology 


Food Science & 
Technology 


0.535 


1 


Plant Food & Human Nutrition 


Food Science & 
Technology 


0.175 


1 




Total: 


9.167 


8 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF '\s the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1996 Journal Citation Report {1991). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Human Nutrition, Foods, and Exercise, Virginia Polytechnic Institute and State 
University. Data is for 1994, and is fi"om the Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1997. 1996JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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1995 Journal Set Data 

Table E13. Journal set data for the Department of Food Science and Human Nutrition, 
University of Florida, 1995. 



Journal Title 


Field Name 


ISIJIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


14.629 


10 


Journal of Biological 
Chemistry 


Biochemistry & Molecular 
Biology 


6.963 


1 


Investigative Ophthalmology & 
Visual Science 


Ophthalmology 


5.250 


1 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


3.980 


2 


Applied Environmental 
Microbiology 


Biotechnology & Applied 
Microbiology 


3.336 


2 


Critical Reviews in 
Microbiology 


Microbiology 


2.435 


1 


Mi crob io logy-UK 


Microbiology 


2.307 


1 


Journal of Nutrition 


Nutrition & Dietetics 


2.141 


4 


Medicine & Science in Sport & 
Exercise 


Sport Sciences 


1.851 


1 


Journal of Agricultural & Food 
Chemistry 


Food Science & 
Technology 


1.502 


3 


Journal of Animal Science 


Agriculture, Dairy & 
Animal Science 


1.435 


2 


Nutrition Review 


Nutrition & Dietetics 


1.302 


1 


Journal of Food Protection 


Food Science & 
Technology 


1.288 


1 


American Journal of Enology 
& Viticulture 


Food Science & 
Technology 


1.259 


1 


Journal of Food Science 


Food Science & 
Technology 


1.249 


6 


Journal of Child Neurology 


Pediatrics 


0.918 


1 


American Journal of Human 
Biology 


Biology, Miscellaneous 


0.728 


1 


Food Control 


Food Science & 
Technology 


0.688 


1 
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Table E13. (continued) 



Journal Title 


Field Name 


ISI JIF 


P 


Journal of Food Processing 
Engineering 


Food Science & 
Technology 


0.571 


1 


Cereal Foods World 


Food Science & 
Technology 


0.425 


1 




Total: 


54.257 


42 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1997 Journal Citation Report (1998). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Food Science and Human Nutrition, University of Florida. Data is for 1995, and 
is from the Web of Science (2000), accessed 8 October 2000. 

Source (s): 

Institute for Scientific Information. 1998. 1997 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISl Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia; Institute for Scientific Information. 
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Table E14. Journal set data for the Department of Foods and Nutrition, University of 
Georgia, 1995. 



Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


14.629 


24 


Journal of Nutrition 


Nutrition & Dietetics 


2.141 


2 


Proceedings of the Society for 
Experimental Biology & 
Medicine 


Medicine, Research & 
Experimental 


2.062 


2 


Medicine & Science in Sport & 
Exercise 


Sport Sciences 


1.851 


2 


Regulatory Peptides 


Physiology 


1.841 


2 


Journal of the American 
Dietetic Association 


Nutrition & Dietetics 


1.435 


1 


Journal of Animal Science 


Agriculture, Dairy & 
Animal Science 


1.435 


2 


Biochemical Archives 


Biochemistry & Molecular 
Biology 


0.288 


1 




Total: 


25.682 


36 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1997 Journal Citation Report (1998). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Foods and Nutrition, University of Georgia. Data is for 1995, and is from the Web 
of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1998. 1997 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia; Institute for Scientific Information. 
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Knoxville, 1995. 
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Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


14.629 


13 


Proceedings of the National 
Academy of Science of the 
USA 


Multidisciplinary Studies 


9.040 


1 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


3.980 


1 


Biochemical Journal 


Biochemistry & Molecular 
Biology 


3.579 


1 


Journal of Nutrition 


Nutrition & Dietetics 


2.141 


5 


Lipids 


Nutrition & Dietetics 


1.947 


1 


Alcohol 


Pharmacology & Pharmacy 


1.264 


1 


Journal of Nutrition Education 


Nutrition & Dietetics 


1.042 


1 


Nutrition Metabolism & 
Cardiovascular Disease 


Nutrition & Dietetics 


0.672 


1 




Total: 


38.294 


25 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1997 Journal Citation Report (1998). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Nutrition, University of Tennessee- Knoxville. Data is for 1995, and is from the 
Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1998. 1997 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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Table El 6. Journal set data for the Department of Human Nutrition, Foods, and Exercise, 
Virginia Polytechnic Institute and State University, 1995. 



Journal Title 


Field Name 


ISI JIF 


P 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


3.980 


1 


Journal of Gerontology Series 
A- Biological Sciences & 
Medical Sciences 


Geriatrics & Gerontology 


1.695 


1 


Muscle & Nerve 


Neurosciences 


1.690 


1 


Journal of the American 
Dietetic Association 


Nutrition & Dietetics 


1.435 


2 


Poultry Science 


Agriculture, Dairy & 
Animal Science 


1.183 


1 


International Journal of Sport 
Nutrition 


Nutrition & Dietetics 


1.141 


1 


Journal of Food Biochemistry 


Nutrition & Dietetics 


0.764 


1 


Ecology of Food & Nutrition 


Nutrition & Dietetics 


0.455 


1 


Journal of Food Quality 


Food Science & 
Technology 


0.260 


1 




Total: 


12.603 


10 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1997 Journal Citation Report (1998). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Human Nutrition, Foods, and Exercise, Virginia Polytechnic Institute and State 
University. Data is for 1995, and is fi'omthe Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1998 . 1997 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database . Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia; Institute for Scientific Information. 
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1996 Journal Set Data 



Table E17. Journal set data for the Department of Food Science and Human Nutrition, 
University of Florida, 1996. 



Journal Title 


Field Name 


ISIJIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


13.861 


9 


Proceedings of the National 
Academy of Science of the 
USA 


Multidisciplinary Studies 


9.821 


1 


Journal of Leukocyte Biology 


Immunology 


4.262 


1 


Molecular Medicine Today 


Biochemistry & Molecular 
Biology 


3.458 


1 


Clinical Chemistry 


Medical Laboratory 
Technology 


3.423 


1 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


3.417 


2 


Environmental & Molecular 
Mutagenesis 


Environmental Sciences 


2.259 


1 


Journal of Nutrition 


Nutrition & Dietetics 


2.127 


3 


Trends in Food Science & 
Technology 


Food Science & 
Technology 


1.898 


1 


Journal of Toxicology & 
Environmental Health 


Toxicology 


1.689 


1 


Journal of Nutritional 
Biochemistry 


Nutrition & Dietetics 


1.335 


1 


Toxicology Letters 


Toxicology 


1.303 


1 


Journal of Food Science 


Food Science & 
Technology 


1.207 


7 


Journal of Trace Elements in 
Medicine & Biology 


Biochemistry & Molecular 
Biology 


0.844 


1 


American Journal of Enology 
& Viticulture 


Food Science & 
Technology 


0.797 


2 


Food Research International 


Food Science & 
Technology 


0.477 


1 


Food Control 


Food Science & 
Technology 


0.418 


3 




Total: 


52.596 


37 
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Table El 7. (continued) 

Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISl JIF is the journal impact factor for the journal as foimd in the Institute for Scientific 
Information’s 1998 Journal Citation Report (1999). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Food Science and Human Nutrition, University of Florida. Data is for 1996, and 
is from the Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1999. 1998 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 





Table E18. Journal set data for the Department of Foods and Nutrition, University of 
Georgia, 1996. 



Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


13.861 


21 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


3.417 


2 


Obesity Research 


Nutrition & Dietetics 


2.265 


1 


Journal of Nutrition 


Nutrition & Dietetics 


2.127 


1 


Journal of Applied Physiology 


Sport Sciences 


2.122 


2 


Journal of Animal Science 


Agriculture, Dairy & 
Animal Science 


1.560 


1 


Sports Medicine 


Sport Sciences 


1.397 


1 


Biology of the Neonate 


Pediatrics 


0.784 


1 


Experimental Aging Research 


Geriatrics & Gerontology 


0.477 


2 


Growth Development & Aging 


Geriatrics & Gerontology 


0.364 


1 




Total: 


28.374 


33 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

ISI Field Name is the same as the Subject Category used by the Institute for Scientific 
Information in their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s \99% Journal Citation Report (\999). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Foods and Nutrition, University of Georgia. Data is for 1996, and is from the Web 
of Science (2000), accessed 8 October 2000. 

Source (sj: 

Institute for Scientific Information. 1999. 1998 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 



Table E19. Journal set data for the Department of Nutrition, University of Tennessee- 
Knoxville, 1996. 
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Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


13.861 


6 


American Journal of Clinical 
Nutrition 


Nutrition & Dietetics 


3.417 


1 


Hormone & Metabolic 
Research 


Endocrinology & 
Metabolism 


2.242 


1 


Journal of Nutrition 


Nutrition & Dietetics 


2.127 ‘ 


4 


Journal of the American 
Dietetic Association 


Nutrition & Dietetics 


1.709 


2 


Journal of Nutritional 
Biochemistry 


Nutrition & Dietetics 


1.335 


2 


Journal of Nutritional 
Education 


Nutrition & Dietetics 


0.782 


1 


Postgraduate Medical Journal 


Medicine, General & 
Internal 


0.478 


1 




Total: 


25.951 


18 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1 998 Journal Citation Report (1999). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Nutrition, University of Tennessee- Knoxville. Data is for 1996, and is from the 
Web of Science (2000), accessed 8 October 2000. 

Source(s): 

Institute for Scientific Information. 1999. 1998 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISl Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia; Institute for Scientific Information. 
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Table E20. Journal set data for the Department of Human Nutrition, Foods, and Exercise, 
Virginia Polytechnic Institute and State University, 1996. 



Journal Title 


Field Name 


ISI JIF 


P 


FASEB Journal 


Biochemistry & Molecular 
Biology 


13.861 


4 


Medicine & Science in Sport & 
Exercise 


Sport Sciences 
0 


2.026 


1 


Journal of Agricultural & Food 
Chemistry 


Food Science & 
Technology 


1.434 


1 


Journal of Nutritional 
Biochemistry 


Nutrition & Dietetics 


1.335 


1 


Journal of Nutritional 
Education 


Nutrition & Dietetics 


0.782 


2 


Nutritional Research 


Nutrition & Dietetics 


0.670 


1 


Journal of Food Quality 


Food Science & 
Technology 


0.273 


1 




Total: 


20.381 


11 



Note(s): 

The journals are listed in order of their respective journal impact factors, highest to lowest. 

Field Name is the same as the Subject Category used by the Institute for Scientific Information in 
their Journal Citation Reports. 

ISI JIF is the journal impact factor for the journal as found in the Institute for Scientific 
Information’s 1998 Journal Citation Report (1999). 

P is the total number of papers in a given journal that is authored by at least one member of the 
Department of Human Nutrition, Foods, and Exercise, Virginia Polytechnic Institute and State 
University. Data is for 1996, and is from the Web of Science (2000), accessed 8 October 2000. 

Source (s): 

Institute for Scientific Information. 1999. 1998 JCR: Journal Citation Reports, Science Edition, A 
Bibliometric Analysis of Science Journals in the ISI Database. Philadelphia, Institute for 
Scientific Information Press. 

Web of Science. 2000. [On-line database]. Philadelphia: Institute for Scientific Information. 
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VITA 



Eric Ackermann was bom in Birmingham, Alabama, in 1953, a propitious year 
according to all the omens. He graduated from Shades Valley High School (now defunct 
and bulldozed into a shopping mall) in 1972. After wandering around a bit, he eventually 
graduated in 1978 with a B.A. in History from the University of Alabama in 
Birmingham. Soon thereafter, Ackermann completed his M. A. in History from the same 
august institution in 1985. After several post-graduate adventures, including a stint as a 
field archaeologist in Tidewater Virginia, he went to work in 1992 in the Special 
Collections Department of the University Libraries at Virginia Polytechnic Institute and 
State University in Blacksburg, Virginia. As of the turn of the new millennium, 
Ackermann is still employed in the same Special Collections unit, now part of the 
University Libraries’ Digital Library and Archives, while he awaits the completion of a 
M.S. in Information Sciences from the University of Tennessee- Knoxville, of which this 
thesis, which you hold in your hands, is part of the process. 
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